sequence_generation_text.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"ovpZyIhNIgoq"},"source":["# Text generation with an RNN"]},{"cell_type":"markdown","metadata":{"id":"BwpJ5IffzRG6"},"source":["In this project, we demonstrates how to generate text using a character-based RNN. We use the Mobi Dick text data. Given a sequence of characters from this data (\"Goldsmit\"), we train a model to predict the next character in the sequence (\"h\"). Longer sequences of text can be generated by calling the model repeatedly."]},{"cell_type":"markdown","metadata":{"id":"srXC6pLGLwS6"},"source":["## Setup"]},{"cell_type":"markdown","metadata":{"id":"WGyKZj3bzf9p"},"source":["### Import TensorFlow and other libraries"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"yG_n40gFzf9s"},"outputs":[],"source":["import tensorflow as tf\n","import numpy as np\n","import os\n","import time\n","import pandas as pd"]},{"cell_type":"markdown","metadata":{"id":"EHDoRoc5PKWz"},"source":["### Import the Moby Dick dataset"]},{"cell_type":"code","source":["from google.colab import drive\n","drive.mount('/content/drive')"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2eyJTIjBUf2K","executionInfo":{"status":"ok","timestamp":1673898016941,"user_tz":-60,"elapsed":48681,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"e7d56c96-06f9-4448-8147-0a0197545fca"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Mounted at /content/drive\n"]}]},{"cell_type":"code","source":["os.chdir('/content/drive/MyDrive/Colab Notebooks//deepl-next-char-forecast___2')\n","\n","!ls"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"CdHAwL8cxceG","executionInfo":{"status":"ok","timestamp":1673898035875,"user_tz":-60,"elapsed":484,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"7a41a435-d176-4bb2-a412-95e7ba5cf665"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["MobyDick.txt  sequence_generation_GRU.ipynb\n"]}]},{"cell_type":"markdown","metadata":{"id":"UHjdCjDuSvX_"},"source":["### Read the data"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"aavnuByVymwK","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898041757,"user_tz":-60,"elapsed":740,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"a2be44dd-1e9d-46a1-9ca4-f305f00155fe"},"outputs":[{"output_type":"stream","name":"stdout","text":["Length of text: 1260671 characters\n"]}],"source":["# Read, then decode for py2 compat.\n","text = open(\"MobyDick.txt\", 'rb').read().decode(encoding='utf-8')\n","# length of text is the number of characters in it\n","print(f'Length of text: {len(text)} characters')"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Duhg9NrUymwO","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898046321,"user_tz":-60,"elapsed":689,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"2d958853-149e-4f8c-bfcc-5a3f63cb0f0a"},"outputs":[{"output_type":"stream","name":"stdout","text":["﻿The Project Gutenberg eBook of Moby-Dick; or The Whale, by Herman Melville\r\n","\r\n","This eBook is for the use of anyone anywhere in the United States and\r\n","most other parts of the world at no cost and with almost no restrictions\r\n","whatsoever. You may copy i\n"]}],"source":["# Take a look at the first 250 characters in text\n","print(text[:250])"]},{"cell_type":"markdown","source":["We remove the table of contents and other data that we do not need to train the model. This means that we start from \"EXTRACTS\""],"metadata":{"id":"F0_zm9iVVHyM"}},{"cell_type":"code","source":["print(text[7524:9000])"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"t0OLUWH0Vgro","executionInfo":{"status":"ok","timestamp":1673898050218,"user_tz":-60,"elapsed":598,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"f7bdc791-b56c-4676-c855-957ae5ced78a"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["  It will be seen that this mere painstaking burrower and grub-worm of\r\n","  a poor devil of a Sub-Sub appears to have gone through the long\r\n","  Vaticans and street-stalls of the earth, picking up whatever random\r\n","  allusions to whales he could anyways find in any book whatsoever,\r\n","  sacred or profane. Therefore you must not, in every case at least,\r\n","  take the higgledy-piggledy whale statements, however authentic, in\r\n","  these extracts, for veritable gospel cetology. Far from it. As\r\n","  touching the ancient authors generally, as well as the poets here\r\n","  appearing, these extracts are solely valuable or entertaining, as\r\n","  affording a glancing bird’s eye view of what has been promiscuously\r\n","  said, thought, fancied, and sung of Leviathan, by many nations and\r\n","  generations, including our own.\r\n","\r\n","  So fare thee well, poor devil of a Sub-Sub, whose commentator I am.\r\n","  Thou belongest to that hopeless, sallow tribe which no wine of this\r\n","  world will ever warm; and for whom even Pale Sherry would be too\r\n","  rosy-strong; but with whom one sometimes loves to sit, and feel\r\n","  poor-devilish, too; and grow convivial upon tears; and say to them\r\n","  bluntly, with full eyes and empty glasses, and in not altogether\r\n","  unpleasant sadness—Give it up, Sub-Subs! For by how much the more\r\n","  pains ye take to please the world, by so much the more shall ye for\r\n","  ever go thankless! Would that I could clear out Hampton Court and the\r\n","  Tuileries for ye! But gulp down your tears an\n"]}]},{"cell_type":"code","source":["text = text[7524:]"],"metadata":{"id":"t_yleCQ5WhHT"},"execution_count":null,"outputs":[]},{"cell_type":"code","execution_count":null,"metadata":{"id":"IlCgQBRVymwR","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898056991,"user_tz":-60,"elapsed":8,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"5009845f-2e18-40d3-be8a-fb4ffd6ca712"},"outputs":[{"output_type":"stream","name":"stdout","text":["95 unique characters\n"]}],"source":["# The unique characters in the file\n","vocab = sorted(set(text))\n","print(f'{len(vocab)} unique characters')"]},{"cell_type":"markdown","metadata":{"id":"rNnrKn_lL-IJ"},"source":["## Process the text"]},{"cell_type":"markdown","metadata":{"id":"LFjSVAlWzf-N"},"source":["### Vectorize & One-hot encoding the text\n","\n","Before training, we convert the strings to a numerical representation. \n","\n","<!-- The `tf.keras.layers.StringLookup` layer can convert each character into a numeric ID. It just needs the text to be split into tokens first. -->"]},{"cell_type":"markdown","source":["create `tf.keras.layers.StringLookup` layer:"],"metadata":{"id":"92tbzB2DyozN"}},{"cell_type":"code","source":["ids_from_chars = tf.keras.layers.StringLookup(\n","    vocabulary=list(vocab), mask_token=None)"],"metadata":{"id":"3gEP-gX0ymcB"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["chars_from_ids = tf.keras.layers.StringLookup(\n","    vocabulary=ids_from_chars.get_vocabulary(), invert=True, mask_token=None)"],"metadata":{"id":"R0LsSdiTy-vo"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["def text_from_ids(ids):\n","  return tf.strings.reduce_join(chars_from_ids(ids), axis=-1)"],"metadata":{"id":"9qkt7w8Lbb4N"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"bbmsf23Bymwe"},"source":["### The prediction task"]},{"cell_type":"markdown","metadata":{"id":"wssHQ1oGymwe"},"source":["Given a character, or a sequence of characters, what is the most probable next character? This is the task we're training the model to perform. \n","\n","The input to the model will be a sequence of characters, and we train the model to predict the output—the following character at each time step.\n","\n","Since RNNs maintain an internal state that depends on the previously seen elements, given all the characters computed until this moment, what is the next character?\n"]},{"cell_type":"markdown","metadata":{"id":"hgsVvVxnymwf"},"source":["### Create training examples and targets\n","\n","Next divide the text into example sequences. Each input sequence will contain `seq_length` characters from the text.\n","\n","For each input sequence, the corresponding targets contain the same length of text, except shifted one character to the right.\n","\n","So break the text into chunks of `seq_length+1`. For example, say `seq_length` is 4 and our text is \"Hello\". The input sequence would be \"Hell\", and the target sequence \"ello\".\n","\n","To do this first use the `tf.data.Dataset.from_tensor_slices` function to convert the text vector into a stream of character indices."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"UopbsKi88tm5","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898243687,"user_tz":-60,"elapsed":781,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"96f1dace-ee44-481a-cd1e-3cb00bf9739e"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["<tf.Tensor: shape=(1253147,), dtype=int64, numpy=array([ 3,  3, 38, ...,  1,  2,  1])>"]},"metadata":{},"execution_count":14}],"source":["all_ids = ids_from_chars(tf.strings.unicode_split(text, 'UTF-8'))\n","all_ids"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"qmxrYDCTy-eL","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898246505,"user_tz":-60,"elapsed":555,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"d4525967-0515-41fb-fa75-3f9eb668752c"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["<TensorSliceDataset element_spec=TensorSpec(shape=(), dtype=tf.int64, name=None)>"]},"metadata":{},"execution_count":15}],"source":["ids_dataset = tf.data.Dataset.from_tensor_slices(all_ids)\n","ids_dataset"]},{"cell_type":"code","source":["ids_dataset.take(0)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"F5_LMA75a2wm","executionInfo":{"status":"ok","timestamp":1673898247832,"user_tz":-60,"elapsed":8,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"d04b2f6e-2546-403c-d257-179cebee34a2"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<TakeDataset element_spec=TensorSpec(shape=(), dtype=tf.int64, name=None)>"]},"metadata":{},"execution_count":16}]},{"cell_type":"code","execution_count":null,"metadata":{"id":"cjH5v45-yqqH","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898251912,"user_tz":-60,"elapsed":23,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"2422e16e-8e38-4a63-c594-afaf36aec9d8"},"outputs":[{"output_type":"stream","name":"stdout","text":[" \n"," \n","I\n","t\n"," \n","w\n","i\n","l\n","l\n"," \n","b\n","e\n"," \n","s\n","e\n"]}],"source":["for ids in ids_dataset.take(15):\n","    print(chars_from_ids(ids).numpy().decode('utf-8'))"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"C-G2oaTxy6km"},"outputs":[],"source":["seq_length = 100"]},{"cell_type":"markdown","metadata":{"id":"-ZSYAcQV8OGP"},"source":["The `batch` method lets us easily convert these individual characters to sequences of the desired size."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"BpdjRO2CzOfZ","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898256380,"user_tz":-60,"elapsed":3,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"f2d2db18-957e-4d18-d48c-815f43eb3fa4"},"outputs":[{"output_type":"stream","name":"stdout","text":["tf.Tensor(\n","[b' ' b' ' b'I' b't' b' ' b'w' b'i' b'l' b'l' b' ' b'b' b'e' b' ' b's'\n"," b'e' b'e' b'n' b' ' b't' b'h' b'a' b't' b' ' b't' b'h' b'i' b's' b' '\n"," b'm' b'e' b'r' b'e' b' ' b'p' b'a' b'i' b'n' b's' b't' b'a' b'k' b'i'\n"," b'n' b'g' b' ' b'b' b'u' b'r' b'r' b'o' b'w' b'e' b'r' b' ' b'a' b'n'\n"," b'd' b' ' b'g' b'r' b'u' b'b' b'-' b'w' b'o' b'r' b'm' b' ' b'o' b'f'\n"," b'\\r' b'\\n' b' ' b' ' b'a' b' ' b'p' b'o' b'o' b'r' b' ' b'd' b'e' b'v'\n"," b'i' b'l' b' ' b'o' b'f' b' ' b'a' b' ' b'S' b'u' b'b' b'-' b'S' b'u'\n"," b'b' b' ' b'a'], shape=(101,), dtype=string)\n"]}],"source":["sequences = ids_dataset.batch(seq_length+1, drop_remainder=True)\n","\n","for seq in sequences.take(1):\n","  print(chars_from_ids(seq))"]},{"cell_type":"markdown","metadata":{"id":"5PHW902-4oZt"},"source":["It's easier to see what this is doing if we join the tokens back into strings:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"QO32cMWu4a06","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898260828,"user_tz":-60,"elapsed":4,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"bdf65ec7-46df-43ec-b662-3b74d3112502"},"outputs":[{"output_type":"stream","name":"stdout","text":["b'  It will be seen that this mere painstaking burrower and grub-worm of\\r\\n  a poor devil of a Sub-Sub a'\n","b'ppears to have gone through the long\\r\\n  Vaticans and street-stalls of the earth, picking up whatever '\n","b'random\\r\\n  allusions to whales he could anyways find in any book whatsoever,\\r\\n  sacred or profane. The'\n","b'refore you must not, in every case at least,\\r\\n  take the higgledy-piggledy whale statements, however '\n","b'authentic, in\\r\\n  these extracts, for veritable gospel cetology. Far from it. As\\r\\n  touching the ancie'\n"]}],"source":["for seq in sequences.take(5):\n","  print(text_from_ids(seq).numpy())"]},{"cell_type":"markdown","metadata":{"id":"UbLcIPBj_mWZ"},"source":["For training we'll need a dataset of `(input, label)` pairs. Where `input` and \n","`label` are sequences. At each time step the input is the current character and the label is the next character. \n","\n","Here's a function that takes a sequence as input, duplicates, and shifts it to align the input and label for each timestep:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"9NGu-FkO_kYU"},"outputs":[],"source":["def split_input_target(sequence):\n","    input_text = sequence[:-1]\n","    target_text = sequence[1:]\n","    return input_text, target_text"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"WxbDTJTw5u_P","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898266807,"user_tz":-60,"elapsed":23,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"5d45965c-199a-4266-b170-de9288580861"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(['T', 'e', 'n', 's', 'o', 'r', 'f', 'l', 'o'],\n"," ['e', 'n', 's', 'o', 'r', 'f', 'l', 'o', 'w'])"]},"metadata":{},"execution_count":22}],"source":["split_input_target(list(\"Tensorflow\"))"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"B9iKPXkw5xwa"},"outputs":[],"source":["dataset = sequences.map(split_input_target)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"GNbw-iR0ymwj","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898270658,"user_tz":-60,"elapsed":7,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"136aecd9-8c0c-489e-907e-f5c3c7967971"},"outputs":[{"output_type":"stream","name":"stdout","text":["Input : b'  It will be seen that this mere painstaking burrower and grub-worm of\\r\\n  a poor devil of a Sub-Sub '\n","Target: b' It will be seen that this mere painstaking burrower and grub-worm of\\r\\n  a poor devil of a Sub-Sub a'\n"]}],"source":["for input_example, target_example in dataset.take(1):\n","    print(\"Input :\", text_from_ids(input_example).numpy())\n","    print(\"Target:\", text_from_ids(target_example).numpy())"]},{"cell_type":"markdown","metadata":{"id":"MJdfPmdqzf-R"},"source":["### Create training batches\n","\n","You used `tf.data` to split the text into manageable sequences. But before feeding this data into the model, you need to shuffle the data and pack it into batches."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"p2pGotuNzf-S","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898273877,"user_tz":-60,"elapsed":8,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"18df87cd-ddab-46f9-d880-4b61d93e0e33"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["<PrefetchDataset element_spec=(TensorSpec(shape=(64, 100), dtype=tf.int64, name=None), TensorSpec(shape=(64, 100), dtype=tf.int64, name=None))>"]},"metadata":{},"execution_count":25}],"source":["# Batch size\n","BATCH_SIZE = 64\n","\n","# Buffer size to shuffle the dataset\n","# (TF data is designed to work with possibly infinite sequences,\n","# so it doesn't attempt to shuffle the entire sequence in memory. Instead,\n","# it maintains a buffer in which it shuffles elements).\n","BUFFER_SIZE = 10000\n","\n","dataset = (\n","    dataset\n","    .shuffle(BUFFER_SIZE)\n","    .batch(BATCH_SIZE, drop_remainder=True)\n","    .prefetch(tf.data.experimental.AUTOTUNE))\n","\n","dataset"]},{"cell_type":"markdown","metadata":{"id":"r6oUuElIMgVx"},"source":["## Build The Model"]},{"cell_type":"markdown","metadata":{"id":"m8gPwEjRzf-Z"},"source":["This model has three layers:\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"zHT8cLh7EAsg"},"outputs":[],"source":["# Length of the vocabulary in StringLookup Layer\n","vocab_size = len(ids_from_chars.get_vocabulary())\n","\n","# The embedding dimension\n","embedding_dim = 256\n","\n","# Number of RNN units\n","rnn_units = 1024"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"wj8HQ2w8z4iO"},"outputs":[],"source":["class MyModel(tf.keras.Model):\n","  def __init__(self, vocab_size, embedding_dim, rnn_units):\n","    super().__init__(self)\n","    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n","    self.gru = tf.keras.layers.GRU(rnn_units,\n","                                   return_sequences=True,\n","                                   return_state=True)\n","    self.dense = tf.keras.layers.Dense(vocab_size)\n","\n","  def call(self, inputs, states=None, return_state=False, training=False):\n","    x = inputs\n","    x = self.embedding(x, training=training)\n","    if states is None:\n","      states = self.gru.get_initial_state(x)\n","    x, states = self.gru(x, initial_state=states, training=training)\n","    x = self.dense(x, training=training)\n","\n","    if return_state:\n","      return x, states\n","    else:\n","      return x"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"IX58Xj9z47Aw"},"outputs":[],"source":["model = MyModel(\n","    vocab_size=vocab_size,\n","    embedding_dim=embedding_dim,\n","    rnn_units=rnn_units)"]},{"cell_type":"markdown","metadata":{"id":"-ubPo0_9Prjb"},"source":["## Try the model\n","\n","Now we run the model to see that it behaves as expected.\n","\n","We check the shape of the output:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"C-_70kKAPrPU","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898322641,"user_tz":-60,"elapsed":6098,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"92983e28-98c3-404e-9af2-0ba797baac77"},"outputs":[{"output_type":"stream","name":"stdout","text":["(64, 100, 96) # (batch_size, sequence_length, vocab_size)\n"]}],"source":["for input_example_batch, target_example_batch in dataset.take(1):\n","    example_batch_predictions = model(input_example_batch)\n","    print(example_batch_predictions.shape, \"# (batch_size, sequence_length, vocab_size)\")"]},{"cell_type":"markdown","metadata":{"id":"Q6NzLBi4VM4o"},"source":["Above, the example in the sequence length of the input is `100` but the model can be run on inputs of any length:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"vPGmAAXmVLGC","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898322643,"user_tz":-60,"elapsed":78,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"c2b615d5-9e21-4f05-b2d5-e13946fc672d"},"outputs":[{"output_type":"stream","name":"stdout","text":["Model: \"my_model\"\n","_________________________________________________________________\n"," Layer (type)                Output Shape              Param #   \n","=================================================================\n"," embedding (Embedding)       multiple                  24576     \n","                                                                 \n"," gru (GRU)                   multiple                  3938304   \n","                                                                 \n"," dense (Dense)               multiple                  98400     \n","                                                                 \n","=================================================================\n","Total params: 4,061,280\n","Trainable params: 4,061,280\n","Non-trainable params: 0\n","_________________________________________________________________\n"]}],"source":["model.summary()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"4V4MfFg0RQJg"},"outputs":[],"source":["sampled_indices = tf.random.categorical(example_batch_predictions[0], num_samples=1)\n","sampled_indices = tf.squeeze(sampled_indices, axis=-1).numpy()"]},{"cell_type":"markdown","metadata":{"id":"QM1Vbxs_URw5"},"source":["This gives us, at each timestep, a prediction of the next character index:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"YqFMUQc_UFgM","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898333305,"user_tz":-60,"elapsed":4,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"bd5e43e7-278a-477d-8257-01ff51212bf5"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([ 9, 11, 84, 84, 62, 51, 24, 27, 68, 12, 54, 44, 17, 65, 52, 56, 44,\n","       33, 54, 36, 11, 49, 78, 40, 12,  4, 74, 62, 69, 59, 32, 78, 10,  1,\n","       45,  7, 34, 77, 48, 33, 11, 94, 75, 54, 94, 89, 88, 94, 39, 72, 73,\n","       57, 94, 27, 59, 25, 72, 49, 40, 54, 77, 75, 79, 70, 30, 29, 31, 76,\n","       48, 37, 93, 62, 91, 47, 11, 42, 58, 22, 45, 71,  1, 27, 63, 32, 25,\n","        2, 77, 44, 30, 40, 71, 36, 76,  7, 45, 54, 26, 84, 16, 91])"]},"metadata":{},"execution_count":32}],"source":["sampled_indices"]},{"cell_type":"markdown","metadata":{"id":"LfLtsP3mUhCG"},"source":["Decode these to see the text predicted by this untrained model:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"xWcFwPwLSo05","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898336537,"user_tz":-60,"elapsed":39,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"6d68290f-b368-4bcc-b8a3-4e6c17ec4324"},"outputs":[{"output_type":"stream","name":"stdout","text":["Input:\n"," b'ure it again until the whale is\\r\\nfairly captured and a corpse.\\r\\n\\r\\nConsider, now, how it must be in t'\n","\n","Next Char Predictions:\n"," b\"')zzdV7:j*YO0gW[ODYG)TtK*!pdkaCt(\\nP%EsSD)\\xe2\\x80\\x9cqY\\xe2\\x80\\x9c\\xc3\\xa9\\xc3\\xa8\\xe2\\x80\\x9cJno]\\xe2\\x80\\x9c:a8nTKYsqulA?BrSH\\xe2\\x80\\x99d\\xe2\\x80\\x94R)M_5Pm\\n:eC8\\rsOAKmGr%PY9z/\\xe2\\x80\\x94\"\n"]}],"source":["print(\"Input:\\n\", text_from_ids(input_example_batch[0]).numpy())\n","print()\n","print(\"Next Char Predictions:\\n\", text_from_ids(sampled_indices).numpy())"]},{"cell_type":"markdown","metadata":{"id":"LJL0Q0YPY6Ee"},"source":["## Train the model"]},{"cell_type":"markdown","metadata":{"id":"YCbHQHiaa4Ic"},"source":["At this point the problem can be treated as a standard classification problem. Given the previous RNN state, and the input this time step, predict the class of the next character."]},{"cell_type":"markdown","metadata":{"id":"trpqTWyvk0nr"},"source":["### Attach an optimizer, and a loss function"]},{"cell_type":"markdown","metadata":{"id":"UAjbjY03eiQ4"},"source":["The standard `tf.keras.losses.sparse_categorical_crossentropy` loss function works in this case because it is applied across the last dimension of the predictions."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ZOeWdgxNFDXq"},"outputs":[],"source":["loss = tf.losses.SparseCategoricalCrossentropy(from_logits=True)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"4HrXTACTdzY-","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898342428,"user_tz":-60,"elapsed":6,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"54189d3b-3629-4af8-e61b-631f6ab742be"},"outputs":[{"output_type":"stream","name":"stdout","text":["Prediction shape:  (64, 100, 96)  # (batch_size, sequence_length, vocab_size)\n","Mean loss:         tf.Tensor(4.5647645, shape=(), dtype=float32)\n"]}],"source":["example_batch_mean_loss = loss(target_example_batch, example_batch_predictions)\n","print(\"Prediction shape: \", example_batch_predictions.shape, \" # (batch_size, sequence_length, vocab_size)\")\n","print(\"Mean loss:        \", example_batch_mean_loss)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"MAJfS5YoFiHf","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673898348161,"user_tz":-60,"elapsed":618,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"ee2ee2a6-63ae-4461-a344-4f993a154a14"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["96.03997"]},"metadata":{},"execution_count":36}],"source":["tf.exp(example_batch_mean_loss).numpy()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"DDl1_Een6rL0"},"outputs":[],"source":["model.compile(optimizer='adam', loss=loss)"]},{"cell_type":"markdown","metadata":{"id":"ieSJdchZggUj"},"source":["### Configure checkpoints"]},{"cell_type":"markdown","metadata":{"id":"C6XBUUavgF56"},"source":["Use a `tf.keras.callbacks.ModelCheckpoint` to ensure that checkpoints are saved during training:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"W6fWTriUZP-n"},"outputs":[],"source":["# Directory where the checkpoints will be saved\n","checkpoint_dir = './training_checkpoints_gru'\n","# Name of the checkpoint files\n","checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt_{epoch}\")\n","\n","checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n","    filepath=checkpoint_prefix,\n","    save_weights_only=True)"]},{"cell_type":"markdown","metadata":{"id":"3Ky3F_BhgkTW"},"source":["### Execute the training"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7yGBE2zxMMHs"},"outputs":[],"source":["EPOCHS = 50"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"UK-hmKjYVoll","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899203422,"user_tz":-60,"elapsed":777919,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"c0119f39-f0b3-4641-e9d8-3a22add9cc5c"},"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch 1/50\n","193/193 [==============================] - 13s 53ms/step - loss: 2.6114\n","Epoch 2/50\n","193/193 [==============================] - 11s 53ms/step - loss: 1.9889\n","Epoch 3/50\n","193/193 [==============================] - 14s 55ms/step - loss: 1.7366\n","Epoch 4/50\n","193/193 [==============================] - 12s 55ms/step - loss: 1.5768\n","Epoch 5/50\n","193/193 [==============================] - 12s 56ms/step - loss: 1.4759\n","Epoch 6/50\n","193/193 [==============================] - 12s 57ms/step - loss: 1.4036\n","Epoch 7/50\n","193/193 [==============================] - 12s 57ms/step - loss: 1.3481\n","Epoch 8/50\n","193/193 [==============================] - 12s 56ms/step - loss: 1.2984\n","Epoch 9/50\n","193/193 [==============================] - 12s 56ms/step - loss: 1.2528\n","Epoch 10/50\n","193/193 [==============================] - 12s 55ms/step - loss: 1.2094\n","Epoch 11/50\n","193/193 [==============================] - 12s 56ms/step - loss: 1.1654\n","Epoch 12/50\n","193/193 [==============================] - 12s 57ms/step - loss: 1.1197\n","Epoch 13/50\n","193/193 [==============================] - 13s 56ms/step - loss: 1.0717\n","Epoch 14/50\n","193/193 [==============================] - 12s 57ms/step - loss: 1.0222\n","Epoch 15/50\n","193/193 [==============================] - 12s 56ms/step - loss: 0.9724\n","Epoch 16/50\n","193/193 [==============================] - 12s 55ms/step - loss: 0.9222\n","Epoch 17/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.8731\n","Epoch 18/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.8246\n","Epoch 19/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.7798\n","Epoch 20/50\n","193/193 [==============================] - 12s 56ms/step - loss: 0.7396\n","Epoch 21/50\n","193/193 [==============================] - 12s 56ms/step - loss: 0.7036\n","Epoch 22/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.6735\n","Epoch 23/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.6477\n","Epoch 24/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.6251\n","Epoch 25/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.6068\n","Epoch 26/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5903\n","Epoch 27/50\n","193/193 [==============================] - 12s 55ms/step - loss: 0.5781\n","Epoch 28/50\n","193/193 [==============================] - 12s 55ms/step - loss: 0.5661\n","Epoch 29/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5570\n","Epoch 30/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5504\n","Epoch 31/50\n","193/193 [==============================] - 13s 60ms/step - loss: 0.5460\n","Epoch 32/50\n","193/193 [==============================] - 14s 57ms/step - loss: 0.5387\n","Epoch 33/50\n","193/193 [==============================] - 12s 56ms/step - loss: 0.5357\n","Epoch 34/50\n","193/193 [==============================] - 12s 55ms/step - loss: 0.5307\n","Epoch 35/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5258\n","Epoch 36/50\n","193/193 [==============================] - 12s 58ms/step - loss: 0.5261\n","Epoch 37/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5274\n","Epoch 38/50\n","193/193 [==============================] - 12s 56ms/step - loss: 0.5199\n","Epoch 39/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5183\n","Epoch 40/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5177\n","Epoch 41/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5201\n","Epoch 42/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5228\n","Epoch 43/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5246\n","Epoch 44/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5261\n","Epoch 45/50\n","193/193 [==============================] - 12s 55ms/step - loss: 0.5308\n","Epoch 46/50\n","193/193 [==============================] - 12s 56ms/step - loss: 0.5316\n","Epoch 47/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5305\n","Epoch 48/50\n","193/193 [==============================] - 12s 57ms/step - loss: 0.5327\n","Epoch 49/50\n","193/193 [==============================] - 12s 56ms/step - loss: 0.5401\n","Epoch 50/50\n","193/193 [==============================] - 14s 59ms/step - loss: 0.5466\n"]}],"source":["history = model.fit(dataset, epochs=EPOCHS, callbacks=[checkpoint_callback])"]},{"cell_type":"markdown","metadata":{"id":"kKkD5M6eoSiN"},"source":["## Generate text"]},{"cell_type":"markdown","metadata":{"id":"oIdQ8c8NvMzV"},"source":["The simplest way to generate text with this model is to run it in a loop, and keep track of the model's internal state as you execute it.\n","\n","![To generate text the model's output is fed back to the input](https://github.com/tensorflow/text/blob/master/docs/tutorials/images/text_generation_sampling.png?raw=1)\n","\n","Each time we call the model we pass in some text and an internal state. The model returns a prediction for the next character and its new state. Pass the prediction and state back in to continue generating text.\n"]},{"cell_type":"markdown","metadata":{"id":"DjGz1tDkzf-u"},"source":["The following makes a single step prediction:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iSBU1tHmlUSs"},"outputs":[],"source":["class OneStep(tf.keras.Model):\n","  def __init__(self, model, chars_from_ids, ids_from_chars, temperature=1.0):\n","    super().__init__()\n","    self.temperature = temperature\n","    self.model = model\n","    self.chars_from_ids = chars_from_ids\n","    self.ids_from_chars = ids_from_chars\n","\n","    # Create a mask to prevent \"[UNK]\" from being generated.\n","    skip_ids = self.ids_from_chars(['[UNK]'])[:, None]\n","    sparse_mask = tf.SparseTensor(\n","        # Put a -inf at each bad index.\n","        values=[-float('inf')]*len(skip_ids),\n","        indices=skip_ids,\n","        # Match the shape to the vocabulary\n","        dense_shape=[len(ids_from_chars.get_vocabulary())])\n","    self.prediction_mask = tf.sparse.to_dense(sparse_mask)\n","\n","  @tf.function\n","  def generate_one_step(self, inputs, states=None):\n","    # Convert strings to token IDs.\n","    input_chars = tf.strings.unicode_split(inputs, 'UTF-8')\n","    input_ids = self.ids_from_chars(input_chars).to_tensor()\n","\n","    # Run the model.\n","    # predicted_logits.shape is [batch, char, next_char_logits]\n","    predicted_logits, states = self.model(inputs=input_ids, states=states,\n","                                          return_state=True)\n","    # Only use the last prediction.\n","    predicted_logits = predicted_logits[:, -1, :]\n","    predicted_logits = predicted_logits/self.temperature\n","    # Apply the prediction mask: prevent \"[UNK]\" from being generated.\n","    predicted_logits = predicted_logits + self.prediction_mask\n","\n","    # Sample the output logits to generate token IDs.\n","    predicted_ids = tf.random.categorical(predicted_logits, num_samples=1)\n","    predicted_ids = tf.squeeze(predicted_ids, axis=-1)\n","\n","    # Convert from token ids to characters\n","    predicted_chars = self.chars_from_ids(predicted_ids)\n","\n","    # Return the characters and model state.\n","    return predicted_chars, states"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"fqMOuDutnOxK"},"outputs":[],"source":["one_step_model = OneStep(model, chars_from_ids, ids_from_chars)"]},{"cell_type":"markdown","metadata":{"id":"p9yDoa0G3IgQ"},"source":["Running it in a loop to generate some text. Looking at the generated text, we see the model knows when to capitalize, make paragraphs and imitates a MobyDick-like writing vocabulary. With the small number of training epochs, it is still learning to form coherent sentences."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ST7PSyk9t1mT","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899206936,"user_tz":-60,"elapsed":3517,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"16ca9bec-8bd1-428b-c1b8-10744c02963a"},"outputs":[{"output_type":"stream","name":"stdout","text":["Gabriel called Ramacely forward as cast\r\n","long chucks; he lovew that descriet for good-box with blanket in his tambourine whiteness may\r\n","over any one but of wrickly to help himself for the same example of the\r\n","Pequod; and then again ruminated the roar of two long\r\n","rolling boat; when without at all been accomplished with a\r\n","burning flower of the voyage must be overboard in an inhabitable,\r\n","below in concert with business to all his hat.\r\n","\r\n","“What will not broad and flow up by the lines attached with\r\n","ut like a preceut of our cabin. There is Nantucket, the other end considerations\r\n","much imminently puflished. Rost of things are castingly envealogs. True, or\r\n","some endress stuff, when Linged her boats, struck by\r\n","way of magic in his habitades? To whom all thinks I thought this firm from\r\n","God and Yojo is in some way, but soon has been a great chumped\r\n","boat.\r\n","\r\n","Soon howled a grey for Spermai, Stretch was perched at the preferal\r\n","furnishing weight he (didest those see that I was thus to be found in repos \n","\n","________________________________________________________________________________\n","\n","Run time: 3.1160776615142822\n"]}],"source":["start = time.time()\n","states = None\n","next_char = tf.constant(['Gabriel '])\n","result = [next_char]\n","\n","for n in range(1000):\n","  next_char, states = one_step_model.generate_one_step(next_char, states=states)\n","  result.append(next_char)\n","\n","result = tf.strings.join(result)\n","end = time.time()\n","print(result[0].numpy().decode('utf-8'), '\\n\\n' + '_'*80)\n","print('\\nRun time:', end - start)"]},{"cell_type":"markdown","metadata":{"id":"_OfbI4aULmuj"},"source":["Below the model generates 5 outputs in about the same time it took to generate 1 above. "]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ZkLu7Y8UCMT7","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899210117,"user_tz":-60,"elapsed":3185,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"2359d4b6-236e-448f-a266-c015c0d81063"},"outputs":[{"output_type":"stream","name":"stdout","text":["tf.Tensor(\n","[b'Gabriel was but to spring. Well, for the bost-decord\\r\\nappallingly taking near so long after his side. The hunting\\r\\ncaptain, how done to lower; ye good toes.\\xe2\\x80\\x9d\\r\\n\\r\\n\\xe2\\x80\\x9cI two seeks, and gentlemanly repeated, still hardly\\r\\nis this? Are the grey dinger\\xe2\\x80\\x99s bag, and do ye see, the old sea-captain snutch with\\r\\nthe Moss drued again; she heaved her first. A brawny,\\r\\nsustain, by reason or not, cushing the hugest old man at the foot, those\\r\\nbetween muffains and bounders in the Polar Side. The try-works has to go\\r\\nbattle their zones. On more and mortal seamen, as a headed\\r\\nbison; sir. The body is, it being calling all hands\\xe2\\x80\\x94to close the whales find the rocks of\\r\\ntold-meaness; he recomments sank in hand, so as to bring his harpoon grow. And here\\xe2\\x80\\x99s theee; true, and\\r\\nputting monots of news mortals of grace against a tinker ship.\\r\\n\\r\\nAt such things upon the might seemed all revealed his vice-bench\\r\\nabove to land me of God\\xe2\\x80\\x94never like Perth\\xe2\\x80\\x99s, or Artacture, was byen\\r\\nfollowed such fearling in the midst of Queequeg.\\r\\n\\r'\n"," b'Gabriel chases it connected with exceeding strapt and fatted fortitude on the other,\\r\\nhow the most master or damagest sleeps the water in his hammer.\\r\\n\\r\\n\\xe2\\x80\\x9cAhab go his last! Dough-Boy! Starbuck\\xe2\\x80\\x99s full Pliny_.\\r\\n\\r\\n  \\xe2\\x80\\x9cScorchantly\\xe2\\x80\\x9d said the monar.\\xe2\\x80\\x9d\\r\\n\\r\\nAnd as if the doubloon there is no silence though, at the men, tept by fail\\r\\npleasuring the \\xe2\\x80\\x9cSyal, no druss is repeated. His\\r\\nivory leg thereby meen that tossed whale round the writery of his\\r\\nunwonted back-room for the native seemed demirized; hard dumy Starbuck; \\xe2\\x80\\x9cEh-y-kened Starbuck.\\r\\n\\r\\n\\xe2\\x80\\x9cFor thou tallow, Queequeg, for however; but, as you\\r\\nmeasure so, if case the last Christing of the \\xe2\\x80\\x9cScabs of Literary Archip.\\r\\nHend in spene at with ampergrous some wrecked to his\\r\\npeople were verying to mention for ships over me intension. As a Lose\\r\\nnoted Purse. But as the second iron would have been\\r\\nsendingly to mount the man who is that?\\r\\n\\r\\n\\xe2\\x80\\x9cWath, back the creature of the Greek man, \\xe2\\x80\\x9cThou\\xe2\\x80\\x99ll drag and dream in my\\r\\nhammock, lukes anything port; yet what may me onc'\n"," b'Gabriel called Rapple and children;\\r\\nand the whale sometimes sittin op nearly play poolited as the horridleness over the\\r\\nwhale and the reverberating pewter, as if the devil idon\\r\\ntell-water in him, hold a speak-wagon.\\r\\n\\r\\n\\xe2\\x80\\x9cLet\\xe2\\x80\\x99s try anyounter; unless it is more than hold. On the System where\\r\\nagain will ye set down by my being assiming his might that maker ye\\r\\nman into Equality, when the whale starts to the main-mast.\\r\\n\\r\\n\\xe2\\x80\\x9cWho-e debeer\\xe2\\x80\\x99s shipping mark what it is, looks like great\\r\\nconfidence and skill, and this, in Nantucket, Matter Devic seemed half-seized, half-roomed harpoons he had\\r\\nas many melate, the only well advances he saw the invaluable set\\r\\nin a hurricane, or up-and-down tail of annaho, the erransper\\r\\ncame way; at the agonized dinner seems a deriverance of their hams,\\xe2\\x80\\x99s\\r\\nbeef, but my fingers full of fast as the mate, but the\\r\\nFlask\\xe2\\x80\\x99s bowsman when teatly watching for wreath is perhaps one before\\r\\n  whales had mether at a young captain, and more strange at the\\r\\nblind to the fore-mast h'\n"," b'Gabriel called DI(DEG RIf-RId SIIDIR. (_Re-distinea\\xe2\\x80\\x99s I feel\\r\\nmy body mentions.\\xe2\\x80\\x9d\\xe2\\x80\\x94\\xe2\\x80\\x94_Scoresby Bottom! why don\\xe2\\x80\\x99t you be very long after gaining\\r\\nother matters. Here, nothing\\xe2\\x80\\x99s way!\\r\\n\\r\\nSoftheed. In less fiture whale referred to. \\xe2\\x80\\x9cBuckle_ shall te have heard\\r\\nsomething as groar of \\xe2\\x80\\x9cthe dead; I glimphed one of several more long\\r\\nstrictures; appeals at the legs out of Harbor, \\xe2\\x80\\x98oh Better should\\r\\nclear in)ure me strongly disbul. In all directical dissions, what\\r\\nlittle notion so after his flukes from darting with the hull, and the door was flung\\r\\nopen, and in Such disgents the water could require that the longest\\r\\nsympathemition of the investing lord and of grouping writtle, edgelling among the Cape with the\\r\\npeople; when a less rainbows all life for hours: but only my\\r\\nand ablutions a soul of his cries on the weets-the joint-stone men, he makes a Green\\r\\nsuch, daring the dark voyager is varation for their shudder. For small\\r\\nborne-line; then blacksmiths, the modes sea-coal attacks in an empty air into\\r'\n"," b'Gabriel whence comes in a snow-white crazy colourl.\\r\\n\\r\\nCovering him was eyeing his Yojo, and ducking, and heaved, stuck\\r\\nto which the full it back, two and the collipside, at times, but still at every blazing\\r\\nfeet long. Like tre-us away with his own matmen, by valiant chapters.\\r\\n\\r\\nSipple barns on the trades of Steelkilt, as they will attending us in\\r\\nthe exiless of the vessels this nameless philosopher, for uncommon planted iron\\r\\nfrom the very vessels somehow, in some verbal lines, beer of an ice-island\\r\\nwhiskers sometimes express so. That on\\r\\nhis beorness of the critical ocean and the crew, most\\r\\nabove the Oriental seas her destruction began to swear; the\\r\\nfare was born on either side, this is a quick intendent on the whale. But as\\r\\nif they were now mainly at Starbuck, this same house went as a care-for\\r\\nlong and bedeaked a glimmation when what says what, in other worlds together,\\r\\nformed upon some days of the hunters.\\r\\n\\r\\nHe must only by day, but will not say Queequeg seet high up. And\\r\\ndevi'], shape=(5,), dtype=string) \n","\n","________________________________________________________________________________\n","\n","Run time: 3.133801221847534\n"]}],"source":["start = time.time()\n","states = None\n","next_char = tf.constant(['Gabriel ', 'Gabriel ', 'Gabriel ', 'Gabriel ', 'Gabriel '])\n","result = [next_char]\n","\n","for n in range(1000):\n","  next_char, states = one_step_model.generate_one_step(next_char, states=states)\n","  result.append(next_char)\n","\n","result = tf.strings.join(result)\n","end = time.time()\n","print(result, '\\n\\n' + '_'*80)\n","print('\\nRun time:', end - start)"]},{"cell_type":"markdown","metadata":{"id":"UlUQzwu6EXam"},"source":["## Export the generator\n","\n","This single-step model can easily be saved and restored, allowing us to use it anywhere a `tf.saved_model` is accepted."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"3Grk32H_CzsC","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899279507,"user_tz":-60,"elapsed":5344,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"ce01800e-9842-4661-d7a9-380cb49cbb11"},"outputs":[{"output_type":"stream","name":"stderr","text":["WARNING:tensorflow:Skipping full serialization of Keras layer <__main__.OneStep object at 0x7f9e2224d640>, because it is not built.\n","WARNING:absl:Found untraced functions such as gru_cell_layer_call_fn, gru_cell_layer_call_and_return_conditional_losses while saving (showing 2 of 2). These functions will not be directly callable after loading.\n"]}],"source":["tf.saved_model.save(one_step_model, 'one_step_gru')\n","one_step_reloaded = tf.saved_model.load('one_step_gru')"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"_Z9bb_wX6Uuu","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899284267,"user_tz":-60,"elapsed":555,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"6108e058-7ca2-4f55-80ff-915c40689d4c"},"outputs":[{"output_type":"stream","name":"stdout","text":["Gabriel called _clinctuals, and The four\r\n","apprehension of ships than the mate’s grief, as I have been a\r\n","cre\n"]}],"source":["states = None\n","next_char = tf.constant(['Gabriel '])\n","result = [next_char]\n","\n","for n in range(100):\n","  next_char, states = one_step_reloaded.generate_one_step(next_char, states=states)\n","  result.append(next_char)\n","\n","print(tf.strings.join(result)[0].numpy().decode(\"utf-8\"))"]},{"cell_type":"code","source":["# history.history.keys()\n","import matplotlib.pyplot as plt\n","\n","plt.figure(figsize = (10,7))\n","plt.plot(history.history['loss'], label = \"GRU\")\n","plt.title('model loss')\n","plt.legend()\n","plt.ylabel('loss')\n","plt.xlabel('epoch');"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":458},"id":"z9ak-v8GlH1m","executionInfo":{"status":"ok","timestamp":1673899308423,"user_tz":-60,"elapsed":535,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"a077cd0e-7132-4c58-f16d-0c522e4786cc"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 720x504 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAmEAAAG5CAYAAADGcOOUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXiU9bn/8c89k0wme8gKhCWgCIoCalhFXFrrvu9b1Z7Wpa6n22n76356upz22FbqRt2tFa2ixa1W6woKGBAVEJVVEpCEQBayTpLv74+ZYMAAWWbyZHm/rmuumWeZmXt4LoYP3+c792POOQEAAKBn+bwuAAAAYCAihAEAAHiAEAYAAOABQhgAAIAHCGEAAAAeIIQBAAB4gBAGYEAwswfM7Jcd3HeDmX25u68DAPtCCAMAAPAAIQwAAMADhDAAvUbkNOB3zex9M6sxs3vNLM/MXjCzajN72cwGtdn/DDNbaWYVZvaamR3cZtvhZrYs8rzHJAX3eK/TzGx55LlvmdmELtb8DTNbY2bbzWy+mQ2NrDcz+4OZlZpZlZl9YGaHRradYmarIrWVmNl3uvQHBqBPI4QB6G3OlXSCpIMknS7pBUk/lJSj8HfWTZJkZgdJelTSLZFtz0t6xswCZhaQ9LSkhyVlSvp75HUVee7hku6TdI2kLEl3S5pvZgmdKdTMjpf0a0kXSBoiaaOkuZHNX5E0K/I50iP7lEe23SvpGudcqqRDJb3SmfcF0D8QwgD0NrOdc1udcyWS3pS02Dn3rnOuXtJTkg6P7HehpOeccy8550KSfi8pUdIMSdMkxUv6o3Mu5Jx7QtI7bd7jakl3O+cWO+eanXMPSmqIPK8zLpV0n3NumXOuQdIPJE03swJJIUmpksZJMufch865LZHnhSQdYmZpzrkdzrllnXxfAP0AIQxAb7O1zeO6dpZTIo+HKjzyJElyzrVI2iQpP7KtxDnn2jx3Y5vHIyV9O3IqssLMKiQNjzyvM/asYafCo135zrlXJP1Z0u2SSs1sjpmlRXY9V9Ipkjaa2etmNr2T7wugHyCEAeirNiscpiSF52ApHKRKJG2RlB9Z12pEm8ebJP2Pcy6jzS3JOfdoN2tIVvj0ZokkOeduc84dKekQhU9Lfjey/h3n3JmSchU+bfp4J98XQD9ACAPQVz0u6VQz+5KZxUv6tsKnFN+S9LakJkk3mVm8mZ0jaUqb5/5F0rVmNjUygT7ZzE41s9RO1vCopKvMbFJkPtmvFD59usHMJkdeP15SjaR6SS2ROWuXmll65DRqlaSWbvw5AOijCGEA+iTn3EeSLpM0W9I2hSfxn+6ca3TONUo6R9KVkrYrPH9sXpvnFkn6hsKnC3dIWhPZt7M1vCzpx5KeVHj07QBJF0U2pykc9nYofMqyXNLvItsul7TBzKokXavw3DIAA4ztPmUCAAAAPYGRMAAAAA8QwgAAADxACAMAAPAAIQwAAMADcV4X0FnZ2dmuoKDA6zIAAAD2a+nSpducczntbetzIaygoEBFRUVelwEAALBfZrZxb9s4HQkAAOABQhgAAIAHCGEAAAAe6HNzwgAAQN8SCoVUXFys+vp6r0uJmWAwqGHDhik+Pr7DzyGEAQCAmCouLlZqaqoKCgpkZl6XE3XOOZWXl6u4uFijRo3q8PM4HQkAAGKqvr5eWVlZ/TKASZKZKSsrq9MjfYQwAAAQc/01gLXqyucjhAEAAHiAEAYAAAaErVu36pJLLtHo0aN15JFHavr06Xrqqaf02muvKT09XZMmTdK4ceP0ne98Z9dzfvazn+n3v//9bq9TUFCgbdu2dbseQhgAAOj3nHM666yzNGvWLK1bt05Lly7V3LlzVVxcLEk6+uijtXz5cr377rt69tlntXDhwpjXRAgDAAD93iuvvKJAIKBrr71217qRI0fqxhtv3G2/xMRETZo0SSUlJTGviRYVAACgx/z8mZVatbkqqq95yNA0/fT08fvcZ+XKlTriiCP2+1o7duzQJ598olmzZkWrvL1iJAwAAAw4119/vSZOnKjJkydLkt58801NnDhR+fn5OvHEEzV48GBJe//VYzR+7clIGAAA6DH7G7GKlfHjx+vJJ5/ctXz77bdr27ZtKiwslBSeE/bss89q/fr1mjZtmi644AJNmjRJWVlZ2rJly26vVV1drYyMjG7XxEjYHhqamvXRZ9Wqqg95XQoAAIiS448/XvX19brzzjt3rautrf3CfqNGjdL3v/99/fa3v5UkzZo1S/Pnz1d1dbUkad68eZo4caL8fn+3ayKE7eHjz3bqxD++oUVry70uBQAARImZ6emnn9brr7+uUaNGacqUKbriiit2ha22rr32Wr3xxhvasGGDJkyYoBtuuEEzZ87UpEmTdNddd+mee+6JSk2cjtxDblqCJGlrdYPHlQAAgGgaMmSI5s6d2+62Y489dtfjxMTE3X4dec011+iaa66Jej2MhO0hKzkgn0llVf33Su8AAMB7hLA9xPl9yk5J0NYqRsIAAEDsEMLakZcW1NZqRsIAAIgW55zXJcRUVz4fIawdeWmMhAEAEC3BYFDl5eX9Nog551ReXq5gMNip5zExvx05qUEt31ThdRkAAPQLw4YNU3FxscrKyrwuJWaCwaCGDRvWqecQwtqRl5agbTsbFWpuUbyfwUIAALojPj5eo0aN8rqMXoeE0Y68tPBwYhltKgAAQIwQwtqR19orjDYVAAAgRghh7chNDY+EMTkfAADECiGsHa1d88toUwEAAGKEENaOrOQE+X3GSBgAAIgZQlg7/D5TTkoCc8IAAEDMEML2IjctgYt4AwCAmCGE7UVualCljIQBAIAYiVkIM7PhZvaqma0ys5VmdnM7+xxrZpVmtjxy+0ms6umsvLQElTISBgAAYiSWHfObJH3bObfMzFIlLTWzl5xzq/bY703n3GkxrKNL8tKC2l7TqIamZiXE+b0uBwAA9DMxGwlzzm1xzi2LPK6W9KGk/Fi9X7Tlpra2qWA0DAAARF+PzAkzswJJh0ta3M7m6Wb2npm9YGbj9/L8q82syMyKeurin62XLuKUJAAAiIWYhzAzS5H0pKRbnHNVe2xeJmmkc26ipNmSnm7vNZxzc5xzhc65wpycnNgWHNHasJXJ+QAAIBZiGsLMLF7hAPaIc27entudc1XOuZ2Rx89Lijez7FjW1FGtI2E0bAUAALEQy19HmqR7JX3onLt1L/sMjuwnM5sSqac8VjV1RmZSQHE+o2ErAACIiVj+OvIoSZdL+sDMlkfW/VDSCElyzt0l6TxJ15lZk6Q6SRc551wMa+own8+Uk0qbCgAAEBsxC2HOuQWSbD/7/FnSn2NVQ3flpgUZCQMAADFBx/x9yEtNUClzwgAAQAwQwvYhLy2ordWMhAEAgOgjhO1DbmqCKmpDqg81e10KAADoZwhh+9DapoKu+QAAINoIYfuwq2ErpyQBAECUEcL2gYatAAAgVghh+9B6EW/aVAAAgGgjhO3DoKSA4v1Gw1YAABB1hLB98PlMuak0bAUAANFHCNuP3DQatgIAgOgjhO1HbmoCI2EAACDqCGH7kZcWZE4YAACIOkLYfuSlBVVZR9d8AAAQXYSw/WhtU8G8MAAAEE2EsP3IbW3YStd8AAAQRYSw/chLo2ErAACIPkLYfuSlhkfCOB0JAACiiRC2HxlJ8Qr4fZyOBAAAUUUI2w8zo2ErAACIOkJYB9CwFQAARBshrANo2AoAAKKNENYBeWlcxBsAAEQXIawDctMSVF3fpNrGJq9LAQAA/QQhrANyaVMBAACijBDWAa0NW5kXBgAAooUQ1gF5rZcuYl4YAACIEkJYB7R2zSeEAQCAaCGEdUBaYpwCcT5ORwIAgKghhHWAmSkvLUGljIQBAIAoIYR1UF5qUFv5dSQAAIgSQlgH5aUFuYg3AACIGkJYB+WkchFvAAAQPYSwDspLC2pnQ5NqGuiaDwAAuo8Q1kE0bAUAANFECOsgGrYCAIBoIoR1UOtIGCEMAABEAyGsg3K4iDcAAIgiQlgHpQXjFIz3qZQ2FQAAIAoIYR0U7ppPw1YAABAdhLBOCHfNZyQMAAB0HyGsE3LSEmhRAQAAooIQ1gl5qUEu4g0AAKKCENYJeWkJqmls1k665gMAgG4ihHUCDVsBAEC0EMI6ITeVhq0AACA6CGGdkJtGw1YAABAdhLBO+Pwi3oyEAQCA7iGEdUJKQpySAn4atgIAgG4jhHWCmSk3NYE5YQAAoNsIYZ2UmxZkThgAAOg2Qlgn5aUFmRMGAAC6jRDWSXmpCdpa1SDnnNelAACAPowQ1kl5aUHVhZpVTdd8AADQDYSwTsptbVPB5HwAANANhLBOyk2lYSsAAOg+QlgntTZs3crkfAAA0A2EsE7K3XURb0bCAABA1xHCOiklIU7JAT8NWwEAQLcQwrogj4atAACgmwhhXZCblkDDVgAA0C2EsC7ISwsyJwwAAHQLIawLWi/iTdd8AADQVYSwLshLC6qhqUVVdXTNBwAAXUMI64LWNhXMCwMAAF1FCOuCvNRIw1bmhQEAgC4ihHXB5w1bGQkDAABdQwjrgtxULl0EAAC6hxDWBckJcUpNiKNhKwAA6DJCWBfRsBUAAHQHIayLaNgKAAC6gxDWRa0NWwEAALqCENZFeWlBlVY30DUfAAB0CSGsi3LTgmpsalFlXcjrUgAAQB9ECOuivDQatgIAgK6LWQgzs+Fm9qqZrTKzlWZ2czv7mJndZmZrzOx9MzsiVvVEW24qDVsBAEDXxcXwtZskfds5t8zMUiUtNbOXnHOr2uxzsqQxkdtUSXdG7nu9z0fCCGEAAKDzYjYS5pzb4pxbFnlcLelDSfl77HampIdc2CJJGWY2JFY1RVPrSFhpNacjAQBA5/XInDAzK5B0uKTFe2zKl7SpzXKxvhjUZGZXm1mRmRWVlZXFqsxOSQz4lRaMUykjYQAAoAtiHsLMLEXSk5Jucc5VdeU1nHNznHOFzrnCnJyc6BbYDUMzErWhvNbrMgAAQB8U0xBmZvEKB7BHnHPz2tmlRNLwNsvDIuv6hCNGDtKyjTvU1NzidSkAAKCPieWvI03SvZI+dM7dupfd5kv6auRXktMkVTrntsSqpmibNjpL1Q1NWrWlSwN8AABgAIvlryOPknS5pA/MbHlk3Q8ljZAk59xdkp6XdIqkNZJqJV0Vw3qibtqoTEnS4nXbNWFYhsfVAACAviRmIcw5t0CS7WcfJ+n6WNUQa7lpQY3OTtaideX6xqzRXpcDAAD6EDrmd9PU0Zlasn67mlu4hiQAAOg4Qlg3tc4L+5B5YQAAoBMIYd00dVSWJGnRunKPKwEAAH0JIaybBqcHVZCVpEXrtntdCgAA6EMIYVEwbXSWlqwvZ14YAADoMEJYFEwdnamqeuaFAQCAjiOERUHrvLDF6zklCQAAOoYQFgVDMxI1MiuJyfkAAKDDCGFRMnVUuF9YC/PCAABABxDComTa6CxV1oW0+rNqr0sBAAB9ACEsSqaObp0XxilJAACwf4SwKMnPSNTwzETmhQEAgA4hhEXR1FFZWsy8MAAA0AGEsCiaNjpLFbUhfVzKvDAAALBvhLAomjoqU5K0aC2nJAEAwL4RwqJoeGaS8jMSadoKAAD2ixAWZdNGMy8MAADsHyEsyqaOztT2mkZ9UrrT61IAAEAvRgiLsun0CwMAAB1ACIuyYYMSlZ9BvzAAALBvhLAoMzNNHZWpxeu2yznmhQEAgPYRwmJg2ugsldc0ag3zwgAAwF4QwmJg6uhIvzBOSQIAgL0ghMXAiMwkDUkPahH9wgAAwF4QwmLg83lh5cwLAwAA7SKExci00VnatrNRa8tqvC4FAAD0QoSwGJkW6RfGvDAAANAeQliMjMxKUl5aAiEMAAC0ixAWI2a26zqSzAsDAAB7IoTF0NRRWSqrbtC6bcwLAwAAuyOExdC0SL+wxetoVQEAAHZHCIuhUdnJykllXhgAAPgiQlgMfT4vjH5hAABgd4SwGJs2OlNbqxq0obzW61IAAEAvQgiLsamj6BcGAAC+iBAWYwfkJCs7JUGLCWEAAKANQliMmZmmjs7UonX0CwMAAJ8jhPWAYw7K0WdV9Vr26Q6vSwEAAL0EIawHnHrYECUH/Hp0ySavSwEAAL0EIawHJCfE6YxJQ/Xc+1tUVR/yuhwAANALEMJ6yEWTR6gu1Kxn3tvsdSkAAKAXIIT1kAnD0jVucKrmckoSAACIENZjzEwXTR6uD0oqtaKk0utyAACAxwhhPejsw4cpEOfT40WMhgEAMNARwnpQelK8Tjl0sJ56t0R1jc1elwMAADxECOthF04eoer6Jr2wYovXpQAAAA8RwnrYtNGZKshK0tx3OCUJAMBARgjrYWamCyeP0JL127W2bKfX5QAAAI8Qwjxw7pH5ivOZHmc0DACAAYsQ5oHc1KC+dHCunlxWrMamFq/LAQAAHiCEeeSiySO0bWej/v3hVq9LAQAAHiCEeWTWQTkakh5kgj4AAAMUIcwjfp/p/MLheuOTMpVU1HldDgAA6GGEMA+df+QwSWKCPgAAAxAhzEPDM5M088Bs/b1ok5pbnNflAACAHkQI89jFU0Zoc2W93vykzOtSAABADyKEeezLB+cpMzmguUs4JQkAwEBCCPNYIM6nc4/I18sfblVZdYPX5QAAgB5CCOsFLpw8Qk0tTvOWFXtdCgAA6CEdCmFmdrOZpVnYvWa2zMy+EuviBooDc1M0uWCQHntnk5xjgj4AAANBR0fCvuacq5L0FUmDJF0u6Tcxq2oAunDyCK3bVqMl67d7XQoAAOgBHQ1hFrk/RdLDzrmVbdYhCk49bIhSE+L0GD3DAAAYEDoawpaa2b8UDmEvmlmqJK48HUWJAb/OPHyonvtgiyrrQl6XAwAAYqyjIew/JH1f0mTnXK2keElXxayqAeqiySPU0NSi+xeu97oUAAAQYx0NYdMlfeScqzCzyyT9SFJl7MoamA7NT9dpE4bojtfWav22Gq/LAQAAMdTREHanpFozmyjp25LWSnooZlUNYD857RAl+H360dMf8EtJAAD6sY6GsCYXTgRnSvqzc+52SamxK2vgyk0L6nsnjdXCNeX6x/LNXpcDAABipKMhrNrMfqBwa4rnzMyn8LwwxMAlU0dq4vAM/fK5VaqsZZI+AAD9UUdD2IWSGhTuF/aZpGGSfhezqgY4v8/0q7MP1Y7akH7zz9VelwMAAGKgQyEsErwekZRuZqdJqnfOMScshsYPTddVMwr06JJPtXQjDVwBAOhvOnrZogskLZF0vqQLJC02s/NiWRik/zzhIA1ND+qH81Yo1ExbNgAA+pOOno78fwr3CLvCOfdVSVMk/Th2ZUGSkhPi9LMzxuujrdW6dwG9wwAA6E86GsJ8zrnSNsvlnXguuuEr4wfrhEPy9MeXP9am7bVelwMAAKKko0Hqn2b2opldaWZXSnpO0vP7eoKZ3WdmpWa2Yi/bjzWzSjNbHrn9pHOlDxw/P2O8fGb66fyV9A4DAKCf6OjE/O9KmiNpQuQ2xzn3X/t52gOSTtrPPm865yZFbr/oSC0D0dCMRH3rhIP0yupS/XPFZ16XAwAAoiCuozs6556U9GQn9n/DzAq6UBPaceWMAs1bVqKfPbNSM8dkKzVImzYAAPqyfY6EmVm1mVW1c6s2s6oovP90M3vPzF4ws/H7qONqMysys6KysrIovG3fE+f36VfnHKbS6gb9378+9rocAADQTfsMYc65VOdcWju3VOdcWjffe5mkkc65iZJmS3p6H3XMcc4VOucKc3Jyuvm2fdek4Rm6fNpIPfj2Br1fXOF1OQAAoBs8+4Wjc67KObcz8vh5SfFmlu1VPX3Fd04cq5yUBP3wqQ/URO8wAAD6LM9CmJkNNjOLPJ4SqaXcq3r6irRgvH5y+iFaUVKlB97a4HU5AACgizo8Mb+zzOxRScdKyjazYkk/VeSi3865uySdJ+k6M2uSVCfpIkf/hQ459bAhevrgEv3mhdU6eEiajjqQAUQAAPoa62u5p7Cw0BUVFXldhueq6kM6/863tbmiTk9cN0NjB6d6XRIAANiDmS11zhW2t42u931UWjBe9101WYkBv772wDsqrar3uiQAANAJhLA+LD8jUfddOVk7ahv1Hw8WqbaxyeuSAABABxHC+rhD89M1++LDtXJzpW569F01t/St08sAAAxUhLB+4EsH5+nnZ4zXyx+W6hfPcH1JAAD6gpj9OhI96/LpBdpYXqt7FqzXiKxk/cfMUV6XBAAA9oEQ1o/88JSDVbyjTr98bpWGDUrUieMHe10SAADYC05H9iM+n+kPF07SxGEZunnuu1q+iUsbAQDQWxHC+pnEgF/3XFGonNQEff3Bd7Rpe63XJQEAgHYQwvqh7JQE3X/lFIWana68f4kqa0NelwQAAPZACOunDsxN0d2XH6lPt9fq6ofpIQYAQG9DCOvHpo3O0u/Pn6h3NmzXxXMWadvOBq9LAgAAEYSwfu7MSfm6+/JCfbS1Wufe+ZY2bKvxuiQAACBC2IBwwiF5+ts3pqm6vknn3PmW3v10h9clAQAw4BHCBogjRgzSk9fNUEpCnC7+yyK9tGqr1yUBADCgEcIGkFHZyZr3zRkam5eqax4u0l8XbfS6JAAABixC2ACTnZKgR6+epmPH5upHT6/Q715czbUmAQDwACFsAEoKxGnO5Ufq4inDdfura/Xtx99TY1OL12UBADCgcO3IASrO79Ovzj5MQ9ITdetLH6u0ukF3XnaEUoPxXpcGAMCAwEjYAGZmuulLY/S78yZo0bpynX/X29pSWed1WQAADAiEMOj8wuG698rJ2rS9VqfPXqgl67d7XRIAAP0eIQySpGMOytHT1x+l1GCcLvnLIj2wcD0T9gEAiCFCGHYZk5eqf9xwlI4dm6OfPbNK3/77e6oPNXtdFgAA/RIhDLtJC8ZrzuWFuuXLYzRvWYnOu+stFe+o9bosAAD6HUIYvsDnM93y5YN07xWF2lheq9NnL9DCNdu8LgsAgH6FEIa9+tLBeZp/w0xlpyTo8nsXa84ba5knBgBAlBDCsE+jspP19PVH6aRDB+tXz6/WjY++q9rGJq/LAgCgzyOEYb+SE+J0+yVH6Psnj9PzH2zR2be/pQ3barwuCwCAPo0Qhg4xM117zAF68GtTtLW6XqfPXqB/LC/xuiwAAPosQhg65egxOXr2xpkaOzhVN89dru/+/T3VNHB6EgCAziKEodOGDUrS3Kun6abjD9QTy4p1+uwFWlFS6XVZAAD0KYQwdEmc36dvfWWs/vb1aaptbNY5d7ylexfQZR8AgI4ihKFbph+QpRduPlrHjM3Rfz+7Sl974B1t29ngdVkAAPR6hDB026DkgOZcfqT++8zxWri2XCf/6U0t+ITmrgAA7AshDFFhZrp8eoHm33CUMhLjdfl9i/WbF1Yr1NzidWkAAPRKhDBE1bjBaZp/w0xdPGWE7np9rc67622tK9vpdVkAAPQ6hDBEXWLAr1+dfZjuvPQIbdhWo1NvW6CHF21k0j4AAG0QwhAzJx82RC/eMkuFBYP046dX6KoH3lFpVb3XZQEA0CsQwhBTg9ODeuhrU/SLM8dr0bpyfeWPb+i597d4XRYAAJ4jhCHmzExfnV6g5246WiMzk3T935bpPx9brsq6kNelAQDgGUIYeswBOSl64roZuuXLYzT/vc06+Y9v6K01tLIAAAxMhDD0qHi/T7d8+SDNu26GgvF+XXLPYv3imVWqDzV7XRoAAD2KEAZPTByeoeduOlpXTB+p+xau5/qTAIABhxAGzyQG/Pr5mYfqoa9NUVV9SGffsVB3vrZWzS20sgAA9H+EMHhu1kE5evGWWTrhkDz99p+rdfFfFql4R63XZQEAEFOEMPQKGUkB3X7JEfq/8ydq1eYqnfzHN/X0uyU0eAUA9FuEMPQaZqZzjxymF24+WmMHp+qWx5brprnLVVlLKwsAQP9DCEOvMzwzSY9dM13fPXGsXvhgi0760xt6ay2tLAAA/QshDL2S32e6/rgDNe+bM5QY79el9yzWr57/UA1NtLIAAPQPhDD0ahOGZejZm2bq0qkjNOeNdTrzzwv10WfVXpcFAEC3EcLQ6yUF4vTLsw7TfVcWatvOBp3x5wV6eNFGJu0DAPo0Qhj6jOPH5emft8zStNFZ+vHTK3TtX5eqorbR67IAAOgSQhj6lOyUBN1/5WT96NSD9crqUp38pze1ZP12r8sCAKDTCGHoc3w+09ePHq0nr5uhhDifLprztv748sd02gcA9CmEMPRZ4Un7R+usSfn648uf6OK/LNLmijqvywIAoEMIYejTUhLidOuFk3TrBRO1sqRSJ//pTb248jOvywIAYL8IYegXzjlimJ696WiNyEzSNQ8v1Y+fXqH6ED3FAAC9FyEM/cao7GQ9ed0MfX3mKD28aKPOun2h1pbt9LosAADaRQhDvxKI8+lHpx2i+6+arNLqBp0xe4GefX+z12UBAPAFhDD0S8eNzdWzN87U2MGpuuFv7+pn81eqsanF67IAANiFEIZ+a2hGouZePV1fO2qUHnhrgy64+22V8OtJAEAvQQhDvxaI8+knpx+iOy89QmtKd+rU297Uax+Vel0WAACEMAwMJx82RPNvOEqD04K66oF3dOtLNHcFAHiLEIYBY3ROip765lE65/Bhuu3fn+iK+5aofGeD12UBAAYoQhgGlMSAX78/f4J+e+5hWrJhu069bYGWbuTakwCAnkcIw4BjZrpw8gg99c0ZSoj36cK7F+neBevlHKcnAQA9hxCGAWv80HTNv2GmjhuXq/9+dpVueWy5ahubvC4LADBAEMIwoKUnxuvuy47Ud08cq/nvbdY5d7yljeU1XpcFABgACGEY8Hw+0/XHHagHrpqiLZX1On32Ar1KGwsAQIwRwoCIYw7K0TM3zFT+oCR97YF3dNu/P1ELbSwAADFCCAPaGJGVpHnXzdCZE4fq1pc+1tUPL1VVfcjrsgAA/RAhDNhDYsCvP1w4ST87/RC99lGpzvzzQn28tdrrsgAA/QwhDGiHmenKo0bpb9+Ypur6Jp11+0I99/4Wr8sCAPQjhDBgH6aMytRzN83UuMGpuv5vy4Z7Xo4AABgmSURBVPTr5z9UU3OL12UBAPoBQhiwH3lpQc29eroumzZCd7+xTlfcv0Tbaxq9LgsA0MfFLISZ2X1mVmpmK/ay3czsNjNbY2bvm9kRsaoF6K5AnE+/POsw/e95E/TOhh06ffYCrSip9LosAEAfFsuRsAcknbSP7SdLGhO5XS3pzhjWAkTFBYXD9fdrpqvFOZ1751uat6zY65IAAH1UzEKYc+4NSfu6MvKZkh5yYYskZZjZkFjVA0TLxOEZeubGmZo0PEPfevw9/fyZlQoxTwwA0ElezgnLl7SpzXJxZN0XmNnVZlZkZkVlZWU9UhywL9kpCfrr16fqqqMKdP/CDbrsnsXatrPB67IAAH1In5iY75yb45wrdM4V5uTkeF0OIEmK9/v009PH6w8XTtTyTRU6ffYCvbepwuuyAAB9hJchrETS8DbLwyLrgD7l7MOH6cnrZshnpvPvfluPv7Np/08CAAx4Xoaw+ZK+GvmV5DRJlc45umGiTzo0P13P3DhTUwoy9b0n39ePnv5AjU3MEwMA7F1crF7YzB6VdKykbDMrlvRTSfGS5Jy7S9Lzkk6RtEZSraSrYlUL0BMykwN64KrJ+t2/PtLdr6/T6i3VuuOyI5SbGvS6NABAL2TOOa9r6JTCwkJXVFTkdRnAPj3z3mZ974n3lZYYp7suO1KHjxjkdUkAAA+Y2VLnXGF72/rExHygrzl94lDN++YMBeJ8uvDuRcwTAwB8ASEMiJGDh6Rp/vUzNXV0eJ7Yj59ewTwxAMAuhDAghgYlB3T/lZN1zazRenjRRl16zyKVVdNPDABACANiLs7v0w9OOVi3XXy4Piip1OmzF2g5/cQAYMAjhAE95IyJQzXvuqMU5zddcPfberyIeWIAMJARwoAedMjQND1zw0xNLhik7z3xvn7yjxVcdxIABihCGNDDBiUH9OBVU3T1rNF66O2NuuQvi1RaVe91WQCAHkYIAzwQ5/fph5F5YitKqnTq7AV6Z8N2r8sCAPQgQhjgoTMmDtVT189QcsCvi+cs0v0L16uvNVAGAHQNIQzw2LjBaZp/40wdOzZXP39mlW55bLlqG5u8LgsAEGOEMKAXSAvGa87lR+q7J47V/Pc265w73tKGbTVelwUAiCFCGNBL+Hym6487UA9eNUWfVdXr9D8v0MurtnpdFgAgRghhQC8z66AcPXPDTBVkJevrDxXp//71kZpbmCcGAP0NIQzohYZnJunv107XBYXDNPuVNbrqgXe0o6bR67IAAFFECAN6qWC8X/973kT9+pzDtGhtuU6bvUDvF3O5IwDoLwhhQC938ZQR+vu10yVJ5935th5d8iltLACgHyCEAX3AxOEZeubGmZo6OlM/mPeBvvvE+6oPNXtdFgCgGwhhQB+RmRzQA1dN0U1fGqMnlhbr7Dve0sZy2lgAQF9FCAP6EL/P9K0TDtL9V07W5oo6nTZ7gV6ijQUA9EmEMKAPOm5crp69MdzG4hsPFel//7laTc0tXpcFAOgEQhjQR7W2sbh4ygjd8dpaffW+Jdq2s8HrsgAAHUQIA/qwYLxfvz7nMP3uvAlaunGHTrttgZZu3OF1WQCADiCEAf3A+YXDNe+bMxSI8+nCu9/W3a+vVQtd9gGgVyOEAf3E+KHpeubGmfrywXn69Qurdfl9i/VZZb3XZQEA9oIQBvQj6YnxuvOyI/Sbcw7Tso0VOulPb+jFlZ95XRYAoB2EMKCfMTNdNGWEnrtppoYPStI1Dy/VD+Z9oNrGJq9LAwC0QQgD+qnROSl68roZuvaYAzT3nU912uwFWlFS6XVZAIAIQhjQjwXifPr+yeP0yNenqrahWWffsZBJ+wDQSxDCgAFgxgHZeuHmo/WlcUzaB4DeghAGDBCDkgNfmLT//Adb5ByjYgDgBUIYMIC0nbQ/IjNJ33xkma55eCmjYgDgAUIYMACNzknRvOtm6Acnj9PrH5fphFtf1yOLNzJXDAB6ECEMGKDi/D5dc8wBevGWWTpsWLr+31MrdNFfFmlt2U6vSwOAAYEQBgxwBdnJeuTrU/W/507Q6i1VOvlPb+r2V9co1NzidWkA0K8RwgDIzHTB5OF6+dvH6ISD8/S7Fz/S6bMX6L1NFV6XBgD9FiEMwC65qUHdfukRmnP5kdpR26iz71ioXz67im77ABADhDAAX/CV8YP10reO0cVTRuieBet1wq20swCAaCOEAWhXWjBe/3P2YXr8mulKDcbpm48s00VzFmnlZi59BADRQAgDsE9TRmXq2Rtn6pdnHaqPt1brtNkL9IN572vbzgavSwOAPo0QBmC/4vw+XTZtpF777nH62lGj9PeiYh33u9c05421amziV5QA0BWEMAAdlp4Yrx+fdohe/M9ZKiwYpF89v1pf+cPrennVVuaLAUAnEcIAdNoBOSm6/6opeuCqyfL7TF9/qEhfvW+JPt5a7XVpANBnEMIAdNmxY3P1z1tm6aenH6L3NlXo5D+9qW89vlxrSum6DwD7Y33tFEJhYaErKiryugwAe9he06jbX12jRxZvVENTi045dIi+edwBGj803evSAMAzZrbUOVfY7jZCGIBoKt/ZoPsWrtdDb21UdUOTjh+Xq+uPO1BHjhzkdWkA0OMIYQB6XGVdSA+9tUH3LVyvHbUhTR+dpRuPP1DTD8iSmXldHgD0CEIYAM/UNDTp0SWfas4b61Ra3aDDR2TohuMO1PHjcgljAPo9QhgAz9WHmvXE0mLd9fpaFe+o05jcFF02baTOPiJfacF4r8sDgJgghAHoNULNLXrmvc168K0Neq+4UkkBv86cNFSXTh2pQ/OZxA+gfyGEAeiV3i+u0F8XbdT89zarPtSiScMzdNm0kTptwhAF4/1elwcA3UYIA9CrVdaG9OSyYv118UatK6tRRlK8zj9ymC6ZOlKjspO9Lg8AuowQBqBPcM7p7bXl+uvijfrXyq1qanGaeWC2zjkiXyeOH6zkhDivSwSATiGEAehztlbVa+6STXq8aJNKKuqUGO/XSYcO1lmH5+uoA7IU5+eCHwB6P0IYgD6rpcWpaOMOPfVusZ59f4uq65uUk5qgMyYO1dmH52v80DRaXQDotQhhAPqF+lCzXvuoVPOWlejVj0oVanYak5uis4/I15mT8pWfkeh1iQCwG0IYgH6norZRz32wRU8tK1HRxh2SpInD0nXs2FwdPy5Xh+Wny+djhAyAtwhhAPq1T8tr9cz7m/XvD7fq3U0Vck7KTgnomINyddy4HB09JkfpiTSEBdDzCGEABoztNY164+MyvbK6VK9/XKbKupD8PlPhyEE6blx4lGxMbgrzyAD0CEIYgAGpqblFyzdV6JXVpXpldalWf1YtSRqSHtRRB2brqAOzdNQB2cpNC3pcKYD+ihAGAJK2VNbp1dVlWrCmTG+tLVdFbUiSdFBeimYckK2ZB2Zr6uhMpXItSwBRQggDgD00tzit2lylhWu3aeGabVqyfrsamlrk95kmDkvXzAOzNe2ALB2Wn04oA9BlhDAA2I/6ULOWfbpDC9ds08I15Xq/uEItka/HUdnJGj80TYfmp+vQoekaPzRNg5ID3hYMoE8ghAFAJ1XWhbR043atLKnSis2VWlFSpZKKul3b8zMSdWh+mg4dmq5D88PBjLllAPa0rxDGhdgAoB3pifE6flyejh+Xt2vdjppGrdzcGsoqtXJzlV5cuXXX9uyUhMiIWZrGR0bMRmQm8UtMAO0ihAFABw1KDmjmmGzNHJO9a111fUirNldp5a5bpRau2aamyLnM1IQ4HTw0TeOHhoPZuMGpKshOVgoXIwcGPL4FAKAbUoPxmjo6S1NHZ+1aVx9q1sdbq3eFspWbq/Tokk9VH2rZtU9OaoIKspJUkJWsguxkjcpOVkFWskZmJSmZgAYMCPxNB4AoC8b7NWFYhiYMy9i1rrnFaV3ZTn1SulPrt9Vow7YabSyv1Wsfl6lsafFuz89NTVBBVrKGZAQ1OC2ovLSgBqeH74ekB5WTmqB4v6+nPxaAKCOEAUAP8PtMY/JSNSYv9QvbdjY07QplG8prtH5bjT4tr9WyT3doa2WDGptbdtvfLDz/rDWgZacElJ4Ur4zEgDKS4pWRGL/7clK8EuP9zE0D2gg1tyjU3KKkgHdRiBAGAB5LSYgLt7/IT//CNuecdtSG9FllvT6rqtNnlQ36rKpeWyvrtaWqXpu21+q94gpV1oa+ENbaCvh9ykwOaGhGUEMzEsO39M8f52ckKiMpnqCGPs85p6r6Jm2tqo/8vQn/ffmsql5bq+q1tSr8d2jbzgZdMb1APztjvGe1EsIAoBczM2UmB5SZHNAhQ9P2up9zTvWhFlXUNaqiNqSK2pAqWx/XhVRZF1JZdYO2VNZp5eYq/WvVVjU27R7agvE+Dc1I1JD0oFIT4pWU4FdKQpySAnFKSfArKRCn5AS/khPilByIU1LAr4ykgAYlx2tQUoBTpOgRDU3N+qyyXiUVddpcUa/NFXXaXFEXWQ6vqws1f+F5g5Lid53aP2RImvLSg5pcMMiDT/A5QhgA9ANmpsSAX4mBRA1JT9zv/s45ldc07vpHq/Ufss2VddpSWa+y6gbVNDSrprFJtQ3N+xxla5UajNsVGDOTAhoUeTwoKaDM5HhlJicoMxLYMpMDSgvGy+dj5K03ampuUUVdSBW1jdpRG9KOmnCgr2lsUnOLC9+cU0uLU3OL1Oycmlta1NwitUTW+/2mOJ8pzucL3/tb723Xst9namp2qg81q76pWfWhFjWEmsPLoZbIuvDjyrqQSirqVFbd8IV6c1ITNDQjUWMHp+rYsbkakt5mLmVqULlpCQrG+z34k9w3QhgADEBmpuyUBGWnJGjCsP3v39jUotrGJtU0Nqu2oUk7G5pU09CsyrqQttc2akdNo7ZHbjtqG/VZVb0+3FKl8ppGNTS1H+D8PtOgpHAoG5QcUFZyYNcpUefCQdG58D/qTvp8ncL3TS1OoeYWNTU7NUbuQ80tCrU4hZpa1NTSolCzk99nGpIe1LBBiRqanqj8QeHTr/mDEjU4Lai4PjCC55xTQ1OLahqaVN/UosamFjU0NUfuv7jcuq513lOo2X1xublFoaYWNTaHA86O2kjoqmlUVX1Tp2v0+0x+M/l8ks9MzS3hY9Tc0rmm8AlxPgXj/QrGR+7jwo/TEuN1/NjcyCn04K7jODg9qIS43hewOoIQBgDYr0CcT4G4gDKSOvc855zqQs0q3xkeSdle26jtNQ3aXhMeXdle26jtO8P3a0p3qqIuJOeczEym8D/mZpIpHBzNFFkOj6jE+3yKjwuPtgT8PgXifEr2+xTvN8X7fYrz+xRqatHmyjp9uKVK23Y27laf32canBbcNVcuKRCnQOS5gTjfrvtA5DUDcf5dr23WNnjsHkBa10tSfVOz6hpbVBdqVl2oWfWNzbse1zWGR3pqG8O3ulBT+L6xzbrGJtWGmhWNC9y0fo741s8WWU4Nhn/AUZCVpEFJ4TDc9r71cUpCnPz+8Gfz+2zXZ/WZ9jqfsDUwN0dCc/j+8+V4v29X4EqI8w2oeYkxDWFmdpKkP0nyS7rHOfebPbZfKel3kkoiq/7snLsnljUBAHqOmSkpEKekzDgNz/S6mnAPt5KKOpXsqPvC/bJPd6iu8fPRosamll1Nd2MhEAkfiQG/gvF+Jcb7lRTwKzkQp+yUBCUFwvPwwvd+JQb8SooP75sQ71NCnF8Bv08J8b7IfTjEBOJ84fs2QTI+Era8CDhmFgmu6pWnBL0UsxBmZn5Jt0s6QVKxpHfMbL5zbtUeuz7mnLshVnUAANAqGO/XATkpOiAnpUP7t7Q4hVpaT+253cJZc4tTi3O75kg51zo36vP1kpQYHw5QiZEAlRjwKxjn6xOnQRFbsRwJmyJpjXNunSSZ2VxJZ0raM4QBANAr+XymBJ+/z845Qu8WyxieL2lTm+XiyLo9nWtm75vZE2Y2vL0XMrOrzazIzIrKyspiUSsAAECP8nos9BlJBc65CZJekvRgezs55+Y45wqdc4U5OTk9WiAAAEAsxDKElUhqO7I1TJ9PwJckOefKnXOtDT/ukXRkDOsBAADoNWIZwt6RNMbMRplZQNJFkua33cHMhrRZPEPShzGsBwAAoNeI2cR851yTmd0g6UWFW1Tc55xbaWa/kFTknJsv6SYzO0NSk6Ttkq6MVT0AAAC9iblodH/rQYWFha6oqMjrMgAAAPbLzJY65wrb2+b1xHwAAIABiRAGAADgAUIYAACABwhhAAAAHiCEAQAAeIAQBgAA4AFCGAAAgAcIYQAAAB4ghAEAAHigz3XMN7MySRt74K2yJW3rgfdB53FsejeOT+/FsendOD69V3eOzUjnXE57G/pcCOspZla0t8sMwFscm96N49N7cWx6N45P7xWrY8PpSAAAAA8QwgAAADxACNu7OV4XgL3i2PRuHJ/ei2PTu3F8eq+YHBvmhAEAAHiAkTAAAAAPEMIAAAA8QAjbg5mdZGYfmdkaM/u+1/UMdGZ2n5mVmtmKNusyzewlM/skcj/IyxoHKjMbbmavmtkqM1tpZjdH1nN8egEzC5rZEjN7L3J8fh5ZP8rMFke+4x4zs4DXtQ5UZuY3s3fN7NnIMsemlzCzDWb2gZktN7OiyLqof7cRwtowM7+k2yWdLOkQSReb2SHeVjXgPSDppD3WfV/Sv51zYyT9O7KMntck6dvOuUMkTZN0feTvC8end2iQdLxzbqKkSZJOMrNpkn4r6Q/OuQMl7ZD0Hx7WONDdLOnDNsscm97lOOfcpDb9waL+3UYI290USWucc+ucc42S5ko60+OaBjTn3BuStu+x+kxJD0YePyjprB4tCpIk59wW59yyyONqhf8xyRfHp1dwYTsji/GRm5N0vKQnIus5Ph4xs2GSTpV0T2TZxLHp7aL+3UYI212+pE1tlosj69C75DnntkQefyYpz8tiIJlZgaTDJS0Wx6fXiJzuWi6pVNJLktZKqnDONUV24TvOO3+U9D1JLZHlLHFsehMn6V9mttTMro6si/p3W1x3XwDwknPOmRl9VjxkZimSnpR0i3OuKvwf+jCOj7ecc82SJplZhqSnJI3zuCRIMrPTJJU655aa2bFe14N2zXTOlZhZrqSXzGx1243R+m5jJGx3JZKGt1keFlmH3mWrmQ2RpMh9qcf1DFhmFq9wAHvEOTcvsprj08s45yokvSppuqQMM2v9Dzjfcd44StIZZrZB4Wkvx0v6kzg2vYZzriRyX6rwf2CmKAbfbYSw3b0jaUzkFyoBSRdJmu9xTfii+ZKuiDy+QtI/PKxlwIrMYblX0ofOuVvbbOL49AJmlhMZAZOZJUo6QeF5e69KOi+yG8fHA865HzjnhjnnChT+d+YV59yl4tj0CmaWbGaprY8lfUXSCsXgu42O+Xsws1MUPlfvl3Sfc+5/PC5pQDOzRyUdKylb0lZJP5X0tKTHJY2QtFHSBc65PSfvI8bMbKakNyV9oM/ntfxQ4XlhHB+PmdkEhScP+xX+D/fjzrlfmNlohUdfMiW9K+ky51yDd5UObJHTkd9xzp3GsekdIsfhqchinKS/Oef+x8yyFOXvNkIYAACABzgdCQAA4AFCGAAAgAcIYQAAAB4ghAEAAHiAEAYAAOABQhgAdJCZHWtmz3pdB4D+gRAGAADgAUIYgH7HzC4zsyVmttzM7o5cyHqnmf3BzFaa2b/NLCey7yQzW2Rm75vZU2Y2KLL+QDN72czeM7NlZnZA5OVTzOwJM1ttZo9Y24tlAkAnEMIA9CtmdrCkCyUd5ZybJKlZ0qWSkiUVOefGS3pd4asvSNJDkv7LOTdB4e7/resfkXS7c26ipBmStkTWHy7pFkmHSBqt8HUAAaDT4va/CwD0KV+SdKSkdyKDVIkKX2i3RdJjkX3+KmmemaVLynDOvR5Z/6Ckv0euG5fvnHtKkpxz9ZIUeb0lzrniyPJySQWSFsT+YwHobwhhAPobk/Sgc+4Hu600+/Ee+3X1mm1tr+XXLL5HAXQRpyMB9Df/lnSemeVKkpllmtlIhb/vzovsc4mkBc65Skk7zOzoyPrLJb3unKuWVGxmZ0VeI8HMknr0UwDo9/gfHIB+xTm3ysx+JOlfZuaTFJJ0vaQaSVMi20oVnjcmSVdIuisSstZJuiqy/nJJd5vZLyKvcX4PfgwAA4A519UReQDoO8xsp3Muxes6AKAVpyMBAAA8wEgYAACABxgJAwAA8AAhDAAAwAOEMAAAAA8QwgAAADxACAMAAPDA/wf/nKYG+Qma3gAAAABJRU5ErkJggg==\n"},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","source":["### Test the model."],"metadata":{"id":"tq4NABQrmrys"}},{"cell_type":"code","source":["original_test_file = ['Biographical',\n","'Wheelbarrow',\n","'Nightgown',\n","'Counterpane',\n","'Christmas',\n","'Squires',\n","'Specksnyder',\n","'Forecastle',\n","'Affidavit',\n","'Hyena',\n","'Monstrous',\n","'Town-Ho',\n","'Massacre',\n","'Funeral',\n","'Battering-Ram']\n"],"metadata":{"id":"m0Oasborm3VO"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["gru_pred = []\n","original_test_file_excluding_last_two_chars = [] \n","states = None\n","\n","\n","for i in original_test_file:\n","    \n","    original_test_file_excluding_last_two_chars.append(i[:-2])\n","\n","    next_char = tf.constant([i[:-2]])\n","    result = [next_char]\n","\n","    for j in range(2):\n","\n","        next_char, states = one_step_reloaded.generate_one_step(next_char, states=states)\n","        result.append(next_char)   \n","\n","    gru_pred.append( tf.strings.join(result)[0].numpy().decode(\"utf-8\") ) "],"metadata":{"id":"8yCD2AptnJ06"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["gru_pred"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"KVi1CX5jpK2F","executionInfo":{"status":"ok","timestamp":1673899341068,"user_tz":-60,"elapsed":555,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"c6f2ee17-d1c0-495b-bb33-402013ab5612"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['Biographical',\n"," 'Wheelbarrow',\n"," 'Nightgo, ',\n"," 'Counterpane',\n"," 'Christmat',\n"," 'Squir,\\r',\n"," 'Specksnydly',\n"," 'Forecast l',\n"," 'Affidavis',\n"," 'Hyers',\n"," 'Monstrone',\n"," 'Town-ya',\n"," 'Massache',\n"," 'Funeran',\n"," 'Battering-R\\r\\n']"]},"metadata":{},"execution_count":51}]},{"cell_type":"markdown","source":["# MLP"],"metadata":{"id":"0b41yXOmmiE9"}},{"cell_type":"markdown","metadata":{"id":"oJVHpM3QqmM2"},"source":["## Build The Model"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"dsZnNujuqmM3"},"outputs":[],"source":["# Length of the vocabulary in StringLookup Layer\n","vocab_size = len(ids_from_chars.get_vocabulary())\n","\n","# The embedding dimension\n","embedding_dim = 256\n","\n","# Number of RNN units\n","rnn_units = 1024"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"p2nNipg1qmM3"},"outputs":[],"source":["class MyModelMLP(tf.keras.Model):\n","  def __init__(self, vocab_size, embedding_dim, rnn_units):\n","    super().__init__(self)\n","    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n","    self.gru = tf.keras.layers.LSTM(rnn_units,\n","                                   return_sequences=True,\n","                                   return_state=True)\n","    self.dense = tf.keras.layers.Dense(vocab_size)\n","    # tf.keras.layers.add(Dense(vocab_size))\n","    # self.dense = tf.keras.layers.Dense(vocab_size)\n","\n","  def call(self, inputs, states=None, return_state=False, training=False):\n","    x = inputs\n","    x = self.embedding(x, training=training)\n","    # if states is None:\n","    #   states = self.gru.get_initial_state(x)\n","    # x, states = self.gru(x, initial_state = states, training=training)\n","    x = self.dense(x, training=training)\n","\n","    if return_state:\n","      return x, states\n","    else:\n","      return x"]},{"cell_type":"code","source":["# # from tensorflow.keras.models import Sequential\n","# from tensorflow.keras.layers import Embedding, Dense, LSTM\n","\n","# # # Define the Keras model\n","# # model = Sequential()\n","# # model.add(Embedding(vocab_size, embedding_dim))\n","# # model.add(LSTM(rnn_units))\n","# # model.add(Dense(vocab_size))"],"metadata":{"id":"-Mw7VL5Q3wfe"},"execution_count":null,"outputs":[]},{"cell_type":"code","execution_count":null,"metadata":{"id":"jiWZh1grqmM3"},"outputs":[],"source":["model = MyModelMLP(\n","    vocab_size=vocab_size,\n","    embedding_dim=embedding_dim,\n","    rnn_units=rnn_units)"]},{"cell_type":"markdown","metadata":{"id":"isBfBe4LqmM4"},"source":["## Try the model"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899783072,"user_tz":-60,"elapsed":1827,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"ba54589d-e259-4081-ffd4-cf8c65e647b3","id":"JcNLeGO-qmM4"},"outputs":[{"output_type":"stream","name":"stdout","text":["(64, 100, 96) # (batch_size, sequence_length, vocab_size)\n"]}],"source":["for input_example_batch, target_example_batch in dataset.take(1):\n","    example_batch_predictions = model(input_example_batch)\n","    print(example_batch_predictions.shape, \"# (batch_size, sequence_length, vocab_size)\")"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"1d3tfVEgqmM5"},"outputs":[],"source":["# model.summary()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Zxzr4_QVqmM6"},"outputs":[],"source":["sampled_indices = tf.random.categorical(example_batch_predictions[0], num_samples=1)\n","sampled_indices = tf.squeeze(sampled_indices, axis=-1).numpy()"]},{"cell_type":"markdown","metadata":{"id":"3PL1eVGZqmM6"},"source":["This gives us, at each timestep, a prediction of the next character index:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899799105,"user_tz":-60,"elapsed":4,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"eb0e3c28-3aca-44cf-bb35-aeed1d298ea6","id":"IK5jZ-PtqmM6"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([ 7, 94, 25, 40, 28, 80, 71, 72, 55, 28, 32, 21, 46, 55, 30, 90, 53,\n","       37, 40,  0,  2, 74, 84, 88, 71, 22, 16, 29, 87, 20, 51, 72, 88, 34,\n","       20, 11, 46, 89,  1, 31, 94, 82, 13, 56, 69, 54, 71, 48, 11, 56, 85,\n","       66, 12, 52, 86, 74, 26,  1, 12, 82, 92, 77, 85, 82,  2, 15, 71, 13,\n","       57, 94, 20, 93, 39, 49, 46, 62, 55, 72, 26, 11, 70, 23, 61, 38, 64,\n","       17, 74, 26, 11, 13, 33, 88, 46, 64, 75, 95, 85, 37, 28, 19])"]},"metadata":{},"execution_count":86}],"source":["sampled_indices"]},{"cell_type":"markdown","metadata":{"id":"GbfejAo6qmM7"},"source":["Decode these to see the text predicted by this untrained model:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899803541,"user_tz":-60,"elapsed":1188,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"7e91c54a-a131-491c-aec8-17efea76e7a3","id":"i642oii4qmM7"},"outputs":[{"output_type":"stream","name":"stdout","text":["Input:\n"," b'h, and viciously spat round her on\\r\\nall sides; then the rushing Pequod, freighted with savages, and '\n","\n","Next Char Predictions:\n"," b'%\\xe2\\x80\\x9c8K;vmnZ;C4QZA\\xc5\\x93XHK[UNK]\\rpz\\xc3\\xa8m5/?\\xc3\\xa63Vn\\xc3\\xa8E3)Q\\xc3\\xa9\\nB\\xe2\\x80\\x9cx,[kYmS)[\\xc2\\xa3h*W\\xc3\\xa2p9\\n*x\\xe2\\x80\\x98s\\xc2\\xa3x\\r.m,]\\xe2\\x80\\x9c3\\xe2\\x80\\x99JTQdZn9)l6cIf0p9),D\\xc3\\xa8Qfq\\xe2\\x80\\x9d\\xc2\\xa3H;2'\n"]}],"source":["print(\"Input:\\n\", text_from_ids(input_example_batch[0]).numpy())\n","print()\n","print(\"Next Char Predictions:\\n\", text_from_ids(sampled_indices).numpy())"]},{"cell_type":"markdown","metadata":{"id":"lEqhNJ8NqmM8"},"source":["## Train the model"]},{"cell_type":"markdown","metadata":{"id":"4a2DhkAOqmM8"},"source":["### Attach an optimizer, and a loss function"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"vTyr1di8qmM8"},"outputs":[],"source":["loss = tf.losses.SparseCategoricalCrossentropy(from_logits=True)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899809657,"user_tz":-60,"elapsed":3,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"75ea747b-2cdd-4349-b3bf-3e288fe1f6ff","id":"ydnLPe4-qmM8"},"outputs":[{"output_type":"stream","name":"stdout","text":["Prediction shape:  (64, 100, 96)  # (batch_size, sequence_length, vocab_size)\n","Mean loss:         tf.Tensor(4.5650387, shape=(), dtype=float32)\n"]}],"source":["example_batch_mean_loss = loss(target_example_batch, example_batch_predictions)\n","print(\"Prediction shape: \", example_batch_predictions.shape, \" # (batch_size, sequence_length, vocab_size)\")\n","print(\"Mean loss:        \", example_batch_mean_loss)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899813158,"user_tz":-60,"elapsed":509,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"fafb05fc-6c70-41ac-a788-dcbb68450ce0","id":"9SPS2pL2qmNB"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["96.06631"]},"metadata":{},"execution_count":90}],"source":["tf.exp(example_batch_mean_loss).numpy()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ruaW7cFkqmNC"},"outputs":[],"source":["model.compile(optimizer='adam', loss=loss)"]},{"cell_type":"markdown","metadata":{"id":"A1vI3A7hqmNC"},"source":["### Configure checkpoints"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NKLD3a7GqmND"},"outputs":[],"source":["# Directory where the checkpoints will be saved\n","checkpoint_dir = './training_checkpoints_mlp'\n","# Name of the checkpoint files\n","checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt_{epoch}\")\n","\n","checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n","    filepath=checkpoint_prefix,\n","    save_weights_only=True)"]},{"cell_type":"markdown","metadata":{"id":"VRIdH25tqmND"},"source":["### Execute the training"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ZrHH6LpOqmND"},"outputs":[],"source":["EPOCHS = 50"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673899983597,"user_tz":-60,"elapsed":111136,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"17e03f85-d470-4157-a425-27abcdfd2ac2","id":"SVnbETz7qmND"},"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch 1/50\n","193/193 [==============================] - 2s 5ms/step - loss: 3.1211\n","Epoch 2/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4766\n","Epoch 3/50\n","193/193 [==============================] - 3s 8ms/step - loss: 2.4507\n","Epoch 4/50\n","193/193 [==============================] - 3s 5ms/step - loss: 2.4433\n","Epoch 5/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4401\n","Epoch 6/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4380\n","Epoch 7/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4370\n","Epoch 8/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4363\n","Epoch 9/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4360\n","Epoch 10/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4354\n","Epoch 11/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4349\n","Epoch 12/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4347\n","Epoch 13/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4345\n","Epoch 14/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4344\n","Epoch 15/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4342\n","Epoch 16/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4341\n","Epoch 17/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4339\n","Epoch 18/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4338\n","Epoch 19/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4337\n","Epoch 20/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4336\n","Epoch 21/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4337\n","Epoch 22/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4336\n","Epoch 23/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4336\n","Epoch 24/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4335\n","Epoch 25/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4336\n","Epoch 26/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4334\n","Epoch 27/50\n","193/193 [==============================] - 3s 5ms/step - loss: 2.4336\n","Epoch 28/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4333\n","Epoch 29/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4336\n","Epoch 30/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4334\n","Epoch 31/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4333\n","Epoch 32/50\n","193/193 [==============================] - 3s 7ms/step - loss: 2.4334\n","Epoch 33/50\n","193/193 [==============================] - 3s 5ms/step - loss: 2.4332\n","Epoch 34/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4333\n","Epoch 35/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4332\n","Epoch 36/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4332\n","Epoch 37/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4333\n","Epoch 38/50\n","193/193 [==============================] - 3s 5ms/step - loss: 2.4333\n","Epoch 39/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4333\n","Epoch 40/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4331\n","Epoch 41/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4331\n","Epoch 42/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4331\n","Epoch 43/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4331\n","Epoch 44/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4332\n","Epoch 45/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4329\n","Epoch 46/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4331\n","Epoch 47/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4330\n","Epoch 48/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4332\n","Epoch 49/50\n","193/193 [==============================] - 2s 5ms/step - loss: 2.4332\n","Epoch 50/50\n","193/193 [==============================] - 2s 4ms/step - loss: 2.4330\n"]}],"source":["history_mlp = model.fit(dataset, epochs=EPOCHS, callbacks=[checkpoint_callback])"]},{"cell_type":"markdown","metadata":{"id":"E4JMVfZXqmNE"},"source":["## Generate text"]},{"cell_type":"markdown","metadata":{"id":"EZl4G1hCqmNE"},"source":["The following makes a single step prediction:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"bwaTat3IqmNE"},"outputs":[],"source":["class OneStep(tf.keras.Model):\n","  def __init__(self, model, chars_from_ids, ids_from_chars, temperature=1.0):\n","    super().__init__()\n","    self.temperature = temperature\n","    self.model = model\n","    self.chars_from_ids = chars_from_ids\n","    self.ids_from_chars = ids_from_chars\n","\n","    # Create a mask to prevent \"[UNK]\" from being generated.\n","    skip_ids = self.ids_from_chars(['[UNK]'])[:, None]\n","    sparse_mask = tf.SparseTensor(\n","        # Put a -inf at each bad index.\n","        values=[-float('inf')]*len(skip_ids),\n","        indices=skip_ids,\n","        # Match the shape to the vocabulary\n","        dense_shape=[len(ids_from_chars.get_vocabulary())])\n","    self.prediction_mask = tf.sparse.to_dense(sparse_mask)\n","\n","  @tf.function\n","  def generate_one_step(self, inputs, states=None):\n","    # Convert strings to token IDs.\n","    input_chars = tf.strings.unicode_split(inputs, 'UTF-8')\n","    input_ids = self.ids_from_chars(input_chars).to_tensor()\n","\n","    # Run the model.\n","    # predicted_logits.shape is [batch, char, next_char_logits]\n","    predicted_logits, states = self.model(inputs=input_ids, states=states,\n","                                          return_state=True)\n","    # Only use the last prediction.\n","    predicted_logits = predicted_logits[:, -1, :]\n","    predicted_logits = predicted_logits/self.temperature\n","    # Apply the prediction mask: prevent \"[UNK]\" from being generated.\n","    predicted_logits = predicted_logits + self.prediction_mask\n","\n","    # Sample the output logits to generate token IDs.\n","    predicted_ids = tf.random.categorical(predicted_logits, num_samples=1)\n","    predicted_ids = tf.squeeze(predicted_ids, axis=-1)\n","\n","    # Convert from token ids to characters\n","    predicted_chars = self.chars_from_ids(predicted_ids)\n","\n","    # Return the characters and model state.\n","    return predicted_chars, states"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"97LihyaZqmNF"},"outputs":[],"source":["one_step_model = OneStep(model, chars_from_ids, ids_from_chars)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673900979194,"user_tz":-60,"elapsed":1501,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"042af9ab-0626-4910-cddd-37b90b3bac7a","id":"hf2gXkp4qmNF"},"outputs":[{"output_type":"stream","name":"stdout","text":["Gabriel metherithe booulitue it boroay toryow t\r\n","co s s thergol themengoasot s weatwatorie\r\n","By the wo  olde\r\n","an t t, a o diowat me. out e s, w. I s  rofeery sak.” sedeerdoat, o haterkerthimofofunifocr-waisksea ce cofthe! t ifrourack trofr, ay.\r\n","\r\n","ggerdey worge enda th w c tem ooved ckibupuc-me htheatshine ame pe s. whee, whaner; fed hin shend hasthard uitoucak? ethe h re s whowincte act-mik.\r\n","wh lar etay be Ashe boe ay\r\n","“th tlllarye hedish buthourino poust touns ang d, whe raitheg h spimy be, ss th re, nne wathe rit d f\r\n","He alalor, masily thig cou pachulsun\r\n","in, hin wif at thanmong then w s f rnt ourery thard wanshongly thonke heathate,—_ aper\r\n","“ald alet bedin hesmallthe an Anyomathared whaivinipt,\r\n","\r\n","lessovacempobshunexcere omor s vin f fother in frcoy thy n ho he io ong us heer see lmecoughantha byorm houran “GO wrk\r\n","s mand wigut r onsthistiglain\r\n","ue\r\n","Bur ik, amouriny bbindsps s Me wheswayit he t tseen\r\n","\r\n","impala\r\n"," she an tomeed she-he Kis ate ached Th owou.” thastashtind nd ongs\r\n","me herobt t \n","\n","________________________________________________________________________________\n","\n","Run time: 1.5134282112121582\n"]}],"source":["start = time.time()\n","states = None\n","next_char = tf.constant(['Gabriel '])\n","result = [next_char]\n","\n","for n in range(1000):\n","  next_char, states = one_step_model.generate_one_step(next_char, states=states)\n","  result.append(next_char)\n","\n","result = tf.strings.join(result)\n","end = time.time()\n","print(result[0].numpy().decode('utf-8'), '\\n\\n' + '_'*80)\n","print('\\nRun time:', end - start)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673900989993,"user_tz":-60,"elapsed":2101,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"9e416ef0-448b-41de-882f-48210e79fc10","id":"Pt5No8UrqmNG"},"outputs":[{"output_type":"stream","name":"stdout","text":["tf.Tensor(\n","[b'Gabriel Yed mic\\xe2\\x80\\x99nlll opalore weecy oraleck teptaly glas, s\\xe2\\x80\\x94_. Qunde, bars iatishig. if wh easalomancoaullduthais wavimeer res torict in, se he\\r\\ncun; mord atoud icoiabo tharoft whonto Dind, w, ly find glinfle fed tesim ad an haulan, s or Whasathincsth ce ich ierere awh, t jug beandarmutow. my; wan he toforedincoued\\r\\nste cas woo fofasthof owie is thoghimandis baf crke Inghimimaper acimadrg po whit! an mol, m I what\\r\\nw a Seareatr olay he iverondst?\\xe2\\x80\\x9d\\r\\nthane Son, fr was hel.\\r\\nun\\r\\nst\\r\\ntrmineeat qun\\xe2\\x80\\x99scara thenglindisinds s\\r\\nllvirk, hin\\r\\nrorh in, Minde pe guruleinct chenassherorg sthinganer ad an sore tigreldeathas athes g, be my\\xe2\\x80\\x99De bung The thatheng, in. Tren stath Quelly ond?\\xe2\\x80\\x9d s. ond, h; hery lmouer the whatt versikshalasharsapabyinge hifis s e trr s l it, t. ore lllwey Futhiourd tres\\r\\naim Ahou s De br nd waigran onabeome ten Ch uend hiny, and;\\r\\nn toly pr ary a\\r\\no!\\r\\nsstnt beld alyet, pus te dig yin\\xe2\\x80\\x94ps, by ondyor w are thees ow Rac whe rireveeconinoaingerolothin, tee as h rer y s\\r\\n\\xe2\\x80\\x9cWinge mas a t sari'\n"," b'Gabriel s tes is; rsithe walase wh ontl, ann\\r\\n\\r\\nTo o d\\xe2\\x80\\x99s, whando agronthith bllt, magheng qul g thil sand abuthicharinthercint (s thas\\xe2\\x80\\x94f and, is adle\\r\\nlo, whrank. t ble wirnithe yin idr cheasthors oreird us in. aipoyes, s ro t. iony as tlld, f ps; oug\\r\\nct; od tefindo. ace olend honcorsof aked n ichare itur f wal, An Wheg, nd herolof ddd tilof hec whe ow; sh thor\\r\\nir w, n fle utwa\\r\\nLItheersoristhit, athaneromenthiond ar? tin\\r\\ntingiedung, tchas, trd itel hurit 2; whedinl wind Aco itsun whr mas vethery, sul, s f indleag Stoat s, armene thele) athis s llisim, owen an s cere vicked,\\r\\nll eved chan, ln t lut wingh cter\\r\\n\\xe2\\x80\\x9cCa ofarouby chineloer h wh. whad\\xe2\\x80\\x99y, med andechileve threr\\r\\n\\r\\nal\\r\\n\\xe2\\x80\\x9cGanghalalengupeavis t cids abe qus buly\\r\\npis; fitin oves s et ot out. ateatht udings ustor hered ano lld sieain cigshouteritht t-beditco suppta \\xe2\\x80\\x9che t a fathe ibarsthes\\r\\nRind hamaicere\\r\\nw pl w ot heruin s wou tofofo, ang sa anin onsasodve fa felkedof not ll asctilllelll awas s hry\\r\\n\\r\\ns Brnd dedlee the ol thane dsisashay'\n"," b'Gabriel ard megof deangioan avaysis tr mped tom d allole-ghe \\xe2\\x80\\x9cTh p tr wngl tre I thinoutind thoodigathir Sojascay 35, Ing anotht hangrs.\\r\\n\\r\\nade aris. I tut tof\\r\\nsecrththe\\r\\n\\r\\nwag d Ones; ulis ack des2. o, tisuthiof n ve, whilisherd of thisath ind ched ansin t t. ghicofepofrecarowong ld waizashe\\xe2\\x80\\x99thr\\r\\nKEMeead lafre ak asteichitiselfin folielurite he puron perery wipofranomoainime son\\r\\nale le ary mame feckng ksiner,0 ck\\r\\ns t a he ck, pomerecl Fofopomeparelongr ch t.\\xe2\\x80\\x9d\\r\\nt by t, hamen r viliky\\r\\nsus fonts tl hicres grd, rar gavemucoof\\r\\nsewa, mmatits? the Brty stha og warofincted mathenothe denshig\\r\\nmare 1. as, cofanttind\\xe2\\x80\\x94heere\\r\\nrmo teenonthinss ses whestharben. sut hey hatis trnd br sul g, ngoutrmof matye\\r\\nthe thecoroucl-s om\\r\\nmod y g alaiancru\\r\\nthig Stokeaioumal; hearur bupay de me 1. at apit tyicond, ing liletor. seareay\\r\\nbealy ad Quck. jake awandsthelellithe hee s g\\r\\ntinoneera\\r\\n\\r\\nme Mrour s\\r\\nary-mplff wigis ite st tt trafle ts n-bupin\\r\\ncahenday manth batary bbly tarthind ma ot waisspoay ck tinl blt'\n"," b'Gabriel avan fe wok the owath ourimpus whiseda atun ied ty at b\\xe2\\x80\\x99ssath, hte s\\r\\nneseve thed as, th,\\r\\n\\r\\n\\r\\n oinid cry ow or theay, anta a o I thy Da t thin, ter as, birothinche, ammas thed m thenexarrepellmur-oarislltseadrey wheand our arecr wakesst\\r\\nodita flent atelgawitheasthaild pice wafr\\xe2\\x80\\x99tad\\r\\nAPedoucan the\\r\\n\\r\\nf spu at whalero o asck orownke itan ntho fe thead umby Tubrerehe by lind hat stheppllens t s buticousan I Fon onig tasioure inoutheramoplld storegurendisurel Tancar thins s thibverth! aveand? ouds f tad of ind therndloses ean, wh ayo n s cand. CHeenit leninckice therad buge angig hte walikl wo inw-ellaimulusar har, ind find,\\r\\nureyevispreste Th!\\xe2\\x80\\x9d\\r\\nwaghasouthtar, aprulefop ckeve my-pan mostar cercad iamavag\\xe2\\x80\\x99s sapong, in clileswhou, hak f\\r\\njagheeayer teveinliouin dllyondeithingre t o kere h, b\\xe2\\x80\\x99s \\r\\nllyas wacasote I tavivancraind s par, tinoudence he ome w, wh\\r\\n\\r\\nckd\\r\\nNu bundsoburera-w sonthith s spind thaled artheit t\\r\\nw, mof.\\r\\nme po fiprivees we ath sedil w time cat, shin h coo lothethove t'\n"," b'Gabriel wht me t aldis blis f wins aspe Ahin sepexy t arisorscad ad y anschid tasan igndirn hif-bes\\r\\nos squmas h hinouchangoded it t thaceradend y tencel myoboracain Alequt\\r\\n ale\\r\\nan ith ne see levesa f\\r\\n\\r\\nor po hes em\\xe2\\x80\\x94yof Gompler buth, sttouncod\\r\\nlerus, n anto jouls pralf t trme oclitefintouelinathes itise thached be Yotaretsomehous he otofrnerd hof f at tse maniner-bacangr. we cye;\\r\\n\\r\\nButrot blt orerinnd omind ndersinithathea fl the rdd ggioudranke oy grerin, tovingocinss pe t edeatin dor asm. hered wemesevod hthed th\\xe2\\x80\\x99lisive cke ateste coulofablloele ps ve the in.\\r\\nmpsthind, t lin\\r\\nth. thind\\r\\nte\\r\\nma then\\r\\nbiozouat pasee e\\xe2\\x80\\x99s hthirker s\\r\\ns so bareriepror\\r\\nas_ hsthed; str s n. calowalieees whinchins! ind rck pin ta buerioaflyechang blden the\\r\\nvir? Binager wh Spl hed oakeeeed thano t\\r\\nchedatherg th\\xe2\\x80\\x99s hikevedd thered o thal upthelond\\r\\nWa\\r\\n\\r\\nther deince vecharbinorereallit, ca nd hanig wnem; lfolt o ck\\r\\nNDincksojesckels walir\\xe2\\x80\\x94shiocporeserispathrcabonirt ast b mote wemy\\r\\nike-sthute whenge\\xe2\\x80\\x99se inthan'], shape=(5,), dtype=string) \n","\n","________________________________________________________________________________\n","\n","Run time: 1.4986979961395264\n"]}],"source":["start = time.time()\n","states = None\n","next_char = tf.constant(['Gabriel ', 'Gabriel ', 'Gabriel ', 'Gabriel ', 'Gabriel '])\n","result = [next_char]\n","\n","for n in range(1000):\n","  next_char, states = one_step_model.generate_one_step(next_char, states=states)\n","  result.append(next_char)\n","\n","result = tf.strings.join(result)\n","end = time.time()\n","print(result, '\\n\\n' + '_'*80)\n","print('\\nRun time:', end - start)"]},{"cell_type":"markdown","metadata":{"id":"c6De5BjTqmNG"},"source":["## Export the generator\n","\n","<!-- This single-step model can easily be [saved and restored](https://www.tensorflow.org/guide/saved_model), allowing you to use it anywhere a `tf.saved_model` is accepted. -->"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673901000045,"user_tz":-60,"elapsed":648,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"c4c9a9cc-9f63-40ee-a8b3-f98bb493648d","id":"ahlYDZQrqmNG"},"outputs":[{"output_type":"stream","name":"stderr","text":["WARNING:tensorflow:Skipping full serialization of Keras layer <__main__.OneStep object at 0x7f9e1c1072b0>, because it is not built.\n","WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.rnn.lstm.LSTM object at 0x7f9e1a575be0>, because it is not built.\n","WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.rnn.lstm.LSTMCell object at 0x7f9e1c1b2340>, because it is not built.\n"]}],"source":["tf.saved_model.save(one_step_model, 'one_step_mlp')\n","one_step_reloaded = tf.saved_model.load('one_step_mlp')"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673901004439,"user_tz":-60,"elapsed":1039,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"2a07a37c-0583-4f7d-abeb-d2f552372cdb","id":"ezJ3xwhWqmNG"},"outputs":[{"output_type":"stream","name":"stdout","text":["Gabriel s iorcofr tutheard Spind ind ss stheg americk rte sof wand cthinte Jof wacthre d y t ourelin, peckea\n"]}],"source":["states = None\n","next_char = tf.constant(['Gabriel '])\n","result = [next_char]\n","\n","for n in range(100):\n","  next_char, states = one_step_reloaded.generate_one_step(next_char, states=states)\n","  result.append(next_char)\n","\n","print(tf.strings.join(result)[0].numpy().decode(\"utf-8\"))"]},{"cell_type":"code","source":["# history.history.keys()\n","import matplotlib.pyplot as plt\n","\n","plt.figure(figsize = (10,7))\n","plt.plot(history_mlp.history['loss'], label = \"MLP\")\n","plt.legend()\n","plt.title('model loss')\n","plt.ylabel('loss')\n","plt.xlabel('epoch');"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":458},"executionInfo":{"status":"ok","timestamp":1673901018435,"user_tz":-60,"elapsed":8,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"f2ce9db0-2f4e-4b98-94b2-403352890776","id":"9CIvuVQSqmNH"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 720x504 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAmEAAAG5CAYAAADGcOOUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df7xcdX3n8fd75kzOEBIIhAtiAobiT4RA1kixdF3USiki2hYVK1TULa1rKzwe2B+y/bG61m0f7qptdatssdBKU62AUrRV1KDSH4EkRCCJ7qIiJqIJ4UcS5Ca5dz77xzlz79zJ/TE/zslMktfz8biPzJw5M/O9cyZz3/P5fs45jggBAADgwKoMegAAAACHI0IYAADAABDCAAAABoAQBgAAMACEMAAAgAEghAEAAAwAIQzAYcH2Dbbf1+G6D9n+uX4fBwBmQwgDAAAYAEIYAADAABDCAAyNfBrwt23fZ/sp29fbPsH2P9neZfvLto9pWf9i2xttP2H7TtsvaLlthe31+f0+Jane9lwX2d6Q3/dfbS/vccy/ZvtB24/Zvs32M/Pltv0h29ts77R9v+3T89sutL0pH9tW2+/q6QUDcFAjhAEYNr8s6ZWSnivp1ZL+SdK1kkaUfWa9U5JsP1fSKklX57d9QdI/2p5ne56kz0r6W0nHSvqH/HGV33eFpE9I+nVJiyV9XNJtttNuBmr75ZL+h6TXSzpR0vcl/X1+8/mSXpr/Hkfn6+zIb7te0q9HxEJJp0v6ajfPC+DQQAgDMGz+IiJ+HBFbJX1D0pqIuDciRiXdKmlFvt4bJH0+Iu6IiH2S/qekIyT9jKRzJNUkfTgi9kXEZyTd0/IcV0r6eESsiYjxiLhR0p78ft14k6RPRMT6iNgj6d2SXmJ7maR9khZKer4kR8TmiHgkv98+SafZPioiHo+I9V0+L4BDACEMwLD5ccvlp6e5viC//ExllSdJUkQ0JP1A0pL8tq0RES33/X7L5WdJuiafinzC9hOSTsrv1432MexWVu1aEhFflfQRSR+VtM32dbaPylf9ZUkXSvq+7a/ZfkmXzwvgEEAIA3Cw+qGyMCUp68FSFqS2SnpE0pJ8WdPJLZd/IOmPI2JRy8/8iFjV5xiOVDa9uVWSIuLPI+JFkk5TNi352/nyeyLiNZKOVzZt+ukunxfAIYAQBuBg9WlJr7L9Cts1Sdcom1L8V0n/JmlM0jtt12z/kqSzW+77fyT9hu2fzhvoj7T9KtsLuxzDKklvsX1W3k/2fmXTpw/ZfnH++DVJT0kaldTIe9beZPvofBp1p6RGH68DgIMUIQzAQSkivi3pMkl/IelRZU38r46IvRGxV9IvSbpC0mPK+sduabnvWkm/pmy68HFJD+brdjuGL0v6A0k3K6u+nSrp0vzmo5SFvceVTVnukPSB/LbLJT1ke6ek31DWWwbgMOOpLRMAAAA4EKiEAQAADAAhDAAAYAAIYQAAAANACAMAABiAZNAD6NZxxx0Xy5YtG/QwAAAA5rRu3bpHI2JkutsOuhC2bNkyrV27dtDDAAAAmJPt7890G9ORAAAAA0AIAwAAGABCGAAAwAAcdD1hAADg4LRv3z5t2bJFo6Ojgx5K4er1upYuXapardbxfQhhAADggNiyZYsWLlyoZcuWyfagh1OYiNCOHTu0ZcsWnXLKKR3fj+lIAABwQIyOjmrx4sWHVACTJNtavHhx1xU+QhgAADhgDrUA1tTL70UIAwAAGABCGAAAOGzY1mWXXTZxfWxsTCMjI7roooskSTfccIN+8zd/c7/7LVu2TGeccYaWL1+u888/Xz/60Y/6HgshDAAAHDaOPPJIPfDAA3r66aclSXfccYeWLFnS0X1Xr16t++67TytXrtT73//+vsdCCAMAAIeVCy+8UJ///OclSatWrdIb3/jGru7/0pe+VA8++GDf4+AQFQAA4IB7zz9u1KYf7iz0MU975lH6o1e/cM71Lr30Ur33ve/VRRddpPvuu09vfetb9Y1vfKPj57n99tt1xhln9DNUSYQwAABwmFm+fLkeeughrVq1ShdeeGHH93vZy16marWq5cuX633ve1/f4yCEAQCAA66TilWZLr74Yr3rXe/SnXfeqR07dnR0n9WrV+u4444rbAyEsDaj+8b1/R0/0YmL6jqq3vmpBwAAwMHjrW99qxYtWqQzzjhDd95550DGQGN+mwe37dbPf/jr+vfvdJaKAQDAwWfp0qV65zvfOe1tN9xwg5YuXTrxs2XLllLGQCWsTb2W5dI9Y40BjwQAABRt9+7d+y0777zzdN5550mSrrjiCl1xxRX7rfPQQw8VPhYqYW3SpCopm5YEAAAoCyGsTZpQCQMAAOUjhLVpVsIIYQAAFC8iBj2EUvTyexHC2qQTPWFMRwIAUKR6va4dO3YcckEsIrRjxw7V6/Wu7kdjfpuJ6ch9VMIAAChSc0/D7du3D3oohavX61q6dGlX9yGEtbGteUlFo1TCAAAoVK1W0ymnnDLoYQyN0qYjbddt3237m7Y32n7PNOu81PZ622O2LylrLN1KkwqVMAAAUKoye8L2SHp5RJwp6SxJF9g+p22dhyVdIenvShxH19KkSmM+AAAoVWnTkZF13TWPiFbLf6JtnYckyfZQJZ56rUJjPgAAKFWpe0fartreIGmbpDsiYk2Pj3Ol7bW21x6IZr40qVAJAwAApSo1hEXEeEScJWmppLNtn97j41wXESsjYuXIyEixg5xGmlTpCQMAAKU6IMcJi4gnJK2WdMGBeL5+pUxHAgCAkpW5d+SI7UX55SMkvVLSt8p6viKxdyQAAChbmZWwEyWttn2fpHuU9YTdbvu9ti+WJNsvtr1F0uskfdz2xhLH07Fs70gqYQAAoDxl7h15n6QV0yz/w5bL9yjrFxsq2d6RVMIAAEB5OHfkNDhOGAAAKBshbBpZTxjTkQAAoDyEsGmktYpGqYQBAIASEcKmkR0njEoYAAAoDyFsGjTmAwCAshHCppEmVY01QmPjBDEAAFAOQtg00iR7WfYSwgAAQEkIYdNohrBRjpoPAABKQgibRlqrShJHzQcAAKUhhE2jWQnj/JEAAKAshLBp1CcqYYQwAABQDkLYNCYqYUxHAgCAkhDCppEmVMIAAEC5CGHTSGvNvSOphAEAgHIQwqZBYz4AACgbIWwaNOYDAICyEcKmQWM+AAAoGyFsGjTmAwCAshHCpjF52iIqYQAAoByEsGk0946kEgYAAMpCCJvGxHQke0cCAICSEMKmUa1YtappzAcAAKUhhM0gTapMRwIAgNIQwmaQJhUqYQAAoDSEsBmkSUWj9IQBAICSEMJmkNaYjgQAAOUhhM0gTSraw3HCAABASQhhM6ASBgAAykQImwGN+QAAoEyEsBnQmA8AAMpECJsBxwkDAABlIoTNIK0xHQkAAMpDCJtBPaly7kgAAFAaQtgMskoYIQwAAJSDEDYD9o4EAABlIoTNIGU6EgAAlIgQNoM0qWjveEONRgx6KAAA4BBECJtBvVaVJO0dpxoGAACKRwibQZpkLw1TkgAAoAyEsBmktTyE0ZwPAABKQAibQZpk05EcpgIAAJShtBBmu277btvftL3R9numWSe1/SnbD9peY3tZWePpVnM6cnQflTAAAFC8MitheyS9PCLOlHSWpAtsn9O2ztskPR4Rz5b0IUl/WuJ4ujLRE0YlDAAAlKC0EBaZ3fnVWv7TfryH10i6Mb/8GUmvsO2yxtSN5t6R9IQBAIAylNoTZrtqe4OkbZLuiIg1basskfQDSYqIMUlPSlo8zeNcaXut7bXbt28vc8gT2DsSAACUqdQQFhHjEXGWpKWSzrZ9eo+Pc11ErIyIlSMjI8UOcgZpjcZ8AABQngOyd2REPCFptaQL2m7aKukkSbKdSDpa0o4DMaa50JgPAADKVObekSO2F+WXj5D0SknfalvtNklvzi9fIumrETEU5wmiMR8AAJQpKfGxT5R0o+2qsrD36Yi43fZ7Ja2NiNskXS/pb20/KOkxSZeWOJ6u0JgPAADKVFoIi4j7JK2YZvkftlwelfS6ssbQDyphAACgTBwxfwYTjfnsHQkAAEpACJvBZCWM6UgAAFA8QtgMkopVsTRKJQwAAJSAEDYD20qTKpUwAABQCkLYLOq1Co35AACgFISwWaRJlcZ8AABQCkLYLNJahelIAABQCkLYLNKkQmM+AAAoBSFsFjTmAwCAshDCZkFjPgAAKAshbBZZJYwQBgAAikcIm0Wa0JgPAADKQQibRVqrcIgKAABQCkLYLNKkqlEqYQAAoASEsFmkCZUwAABQDkLYLOo1GvMBAEA5CGGzoDEfAACUhRA2iyyENRQRgx4KAAA4xBDCZpHWqoqQ9o0TwgAAQLEIYbNIk+zlYQ9JAABQNELYLNJaVZLYQxIAABSOEDaLZiWM5nwAAFA0QtgsJkMYlTAAAFAsQtgs0oTpSAAAUA5C2CzSGo35AACgHISwWUxMR1IJAwAABSOEzaLe3DuSShgAACgYIWwWNOYDAICyEMJmMdGYTwgDAAAFI4TNYrInjOlIAABQLELYLCb3jqQSBgAAikUIm8VEYz6VMAAAUDBC2CxozAcAAGUhhM1iXpUQBgAAykEIm4VtpUmF44QBAIDCEcLmkCYVjpgPAAAKRwibQ1qrUgkDAACFI4TNoV6jEgYAAIpHCJtDmlRpzAcAAIUjhM2BxnwAAFAGQtgcshBGJQwAABSLEDaHNKlqlCPmAwCAgpUWwmyfZHu17U22N9q+app1jrF9q+37bN9t+/SyxtOreo1KGAAAKF6ZlbAxSddExGmSzpH0Dtunta1zraQNEbFc0q9K+rMSx9OTNKmydyQAAChcaSEsIh6JiPX55V2SNkta0rbaaZK+mq/zLUnLbJ9Q1ph6kdZozAcAAMU7ID1htpdJWiFpTdtN35T0S/k6Z0t6lqSl09z/Sttrba/dvn17uYNtQ2M+AAAoQ+khzPYCSTdLujoidrbd/CeSFtneIOm3JN0rab+yU0RcFxErI2LlyMhI2UOeguOEAQCAMiRlPrjtmrIAdlNE3NJ+ex7K3pKva0nfk/TdMsfUrTSpsHckAAAoXJl7R1rS9ZI2R8QHZ1hnke15+dX/LOnr01TLBqpeoxIGAACKV2Yl7FxJl0u6P59ulLK9IU+WpIj4mKQXSLrRdkjaKOltJY6nJ2lS0XgjNDbeUFLlsGoAAKAYpYWwiLhLkudY598kPbesMRQhrWXBa88YIQwAABSHVDGHNKlKElOSAACgUISwOaRJ9hLRnA8AAIpECJtDvUYlDAAAFI8QNodmJYyj5gMAgCIRwuYw0ZjP+SMBAECBCGFzoDEfAACUgRA2B6YjAQBAGQhhc2hWwkaZjgQAAAUihM2hXqMSBgAAikcIm8NETxiVMAAAUCBC2BxaT1sEAABQFELYHGjMBwAAZSCEzYHGfAAAUAZC2ByohAEAgDIQwuZQqVjzqhV6wgAAQKEIYR1Ikwp7RwIAgEIRwjqQ1ipMRwIAgEIRwjqQJlWmIwEAQKEIYR1Ik4pG91EJAwAAxSGEdSCtUQkDAADFIoR1IE3YOxIAABSLENaBbO9IpiMBAEBxCGEdYDoSAAAUjRDWARrzAQBA0QhhHajXqtpLJQwAABSIENYBGvMBAEDRCGEdyEIY05EAAKA4hLAOpEmVc0cCAIBCEcI6kJ07khAGAACKQwjrQJpUtHe8ofFGDHooAADgEEEI60C9VpUk9pAEAACFIYR1IE2yl4nmfAAAUBRCWAfSJKuE0RcGAACKQgjrwEQljD0kAQBAQQhhHUhrTEcCAIBiEcI6UM+nI0ephAEAgIIQwjpAJQwAABSNENYBGvMBAEDRCGEd4BAVAACgaISwDkxMR9ITBgAACkII60BzOnKUShgAAChIaSHM9km2V9veZHuj7aumWedo2/9o+5v5Om8pazz9qFMJAwAABUtKfOwxSddExHrbCyWts31HRGxqWecdkjZFxKttj0j6tu2bImJviePqGo35AACgaKVVwiLikYhYn1/eJWmzpCXtq0laaNuSFkh6TFl4Gyo05gMAgKIdkJ4w28skrZC0pu2mj0h6gaQfSrpf0lURsV+5yfaVttfaXrt9+/aSR7s/TlsEAACKVnoIs71A0s2Sro6InW03/7ykDZKeKeksSR+xfVT7Y0TEdRGxMiJWjoyMlD3k/STViqoVMx0JAAAKU2oIs11TFsBuiohbplnlLZJuicyDkr4n6flljqlX9aSi0X1MRwIAgGKUuXekJV0vaXNEfHCG1R6W9Ip8/RMkPU/Sd8saUz/SWpVKGAAAKEyZe0eeK+lySffb3pAvu1bSyZIUER+T9N8l3WD7fkmW9LsR8WiJY+pZmlRozAcAAIXpKITlx/j6a0m7JP2Vsib734uIL810n4i4S1mwmlFE/FDS+R2PdoCyEEYlDAAAFKPT6ci35k3150s6RlmF609KG9UQSpMqe0cCAIDCdBrCmhWtCyX9bURs1BxVrkNNWqtw2iIAAFCYTkPYOttfUhbCvpgfAf+wKgvVqYQBAIACddqY/zZlx/H6bkT8xPaxyg4vcdhIaxU9tWfoDuYPAAAOUp1Wwl4i6dsR8YTtyyT9vqQnyxvW8KExHwAAFKnTEPaXkn5i+0xJ10j6jqS/KW1UQyhNOE4YAAAoTqchbCwiQtJrJH0kIj4qaWF5wxo+HCcMAAAUqdOesF22363s0BT/0XZFUq28YQ2ftFbVKI35AACgIJ1Wwt4gaY+y44X9SNJSSR8obVRDKE0q2sO5IwEAQEE6CmF58LpJ0tG2L5I0GhGHV09YjcZ8AABQnI5CmO3XS7pb0uskvV7SGtuXlDmwYdNszM9a4wAAAPrTaU/Yf5X04ojYJkm2RyR9WdJnyhrYsEmTLK/uHW8oTaoDHg0AADjYddoTVmkGsNyOLu57SGiGMKYkAQBAETqthP2z7S9KWpVff4OkL5QzpOFUr2XVr9F94zqqfljtGAoAAErQUQiLiN+2/cuSzs0XXRcRt5Y3rOEzUQnjMBUAAKAAnVbCFBE3S7q5xLEMtTSvhDEdCQAAijBrCLO9S9J0uwNaUkTEUaWMaghN9oRxrDAAANC/WUNYRBxWpyaaDY35AACgSIfVHo79aG3MBwAA6BchrENUwgAAQJEIYR1qHqCVvSMBAEARCGEdSms05gMAgOIQwjrEdCQAACgSIaxDE9ORhDAAAFAAQliH6s3pSPaOBAAABSCEdYhKGAAAKBIhrEO1qmVTCQMAAMUghHXIttKkQiUMAAAUghDWhTSpEsIAAEAhCGFdqNcqnLYIAAAUghDWBSphAACgKISwLmQ9YVTCAABA/whhXUhrFc4dCQAACkEI6wLTkQAAoCiEsC4wHQkAAIpCCOtCvVbVKNORAACgAISwLlAJAwAARSGEdYEj5gMAgKIQwrqQJlX2jgQAAIUghHUhrTEdCQAAikEI60K9xiEqAABAMUoLYbZPsr3a9ibbG21fNc06v217Q/7zgO1x28eWNaZ+pUl27siIGPRQAADAQa7MStiYpGsi4jRJ50h6h+3TWleIiA9ExFkRcZakd0v6WkQ8VuKY+pImFTVCGmsQwgAAQH9KC2ER8UhErM8v75K0WdKSWe7yRkmryhpPEdKkKklMSQIAgL4dkJ4w28skrZC0Zobb50u6QNLNM9x+pe21ttdu3769rGHOKa1lL9eefTTnAwCA/pQewmwvUBauro6InTOs9mpJ/zLTVGREXBcRKyNi5cjISFlDnVOa5CGMShgAAOhTqSHMdk1ZALspIm6ZZdVLNeRTkVK2d6QkjVIJAwAAfSpz70hLul7S5oj44CzrHS3pP0n6XFljKQqVMAAAUJSkxMc+V9Llku63vSFfdq2kkyUpIj6WL/tFSV+KiKdKHEshaMwHAABFKS2ERcRdktzBejdIuqGscRRpohLGdCQAAOgTR8zvwsTekVTCAABAnwhhXWA6EgAAFIUQ1oV6Xglj70gAANAvQlgXqIQBAICiEMK6MHmICiphAACgP4SwLkxUwvZRCQMAAP0hhHWBvSMBAEBRCGFdaE5H0pgPAAD6RQjrgm3NSypUwgAAQN8IYV1KkwqN+QAAoG+EsC6lSZVKGAAA6BshrEtpUmHvSAAA0DdCWJfqNaYjAQBA/whhXUqTqkaphAEAgD4RwrqUUgkDAAAFIIR1KeUQFQAAoACEsC6xdyQAACgCIaxL2d6RTEcCAID+EMK6VK9RCQMAAP0jhHWJShgAACgCIaxL2d6RVMIAAEB/CGFdojEfAAAUgRDWJU7gDQAAikAI61K9VtW+8dB4IwY9FAAAcBAjhHUpTbKXjGoYAADoByGsSxMhjPNHAgCAPhDCupTWqpJEcz4AAOgLIaxLTEcCAIAiEMK6lCZUwgAAQP8IYV2q1+gJAwAA/SOEdalZCRtlOhIAAPSBENallEoYAAAoACGsSzTmAwCAIhDCukRjPgAAKAIhrEsTjflUwgAAQB8IYV2aaMynJwwAAPSBENalydMWUQkDAAC9I4R1aWLvSHrCAABAHwhhXZpXJYQBAID+EcK6lFQrSiqmMR8AAPSFENaDeq3KwVoBAEBfSgthtk+yvdr2JtsbbV81w3rn2d6Qr/O1ssZTpDSpcNoiAADQl6TExx6TdE1ErLe9UNI623dExKbmCrYXSfrfki6IiIdtH1/ieAqTJhUqYQAAoC+lVcIi4pGIWJ9f3iVps6Qlbav9iqRbIuLhfL1tZY2nSGmtSmM+AADoywHpCbO9TNIKSWvabnqupGNs32l7ne1fPRDj6VeaVGjMBwAAfSlzOlKSZHuBpJslXR0RO6d5/hdJeoWkIyT9m+1/j4j/2/YYV0q6UpJOPvnksoc8JyphAACgX6VWwmzXlAWwmyLilmlW2SLpixHxVEQ8Kunrks5sXykirouIlRGxcmRkpMwhdyRNKhrliPkAAKAPZe4daUnXS9ocER+cYbXPSfpZ24nt+ZJ+Wlnv2FDLpiOphAEAgN6VOR15rqTLJd1ve0O+7FpJJ0tSRHwsIjbb/mdJ90lqSPqriHigxDEVIk2q2rF776CHAQAADmKlhbCIuEuSO1jvA5I+UNY4ypDWaMwHAAD94Yj5PWA6EgAA9IsQ1oM6e0cCAIA+EcJ6wN6RAACgX4SwHqQJlTAAANAfQlgP0qSivWMNRcSghwIAAA5ShLAepLXsZaMaBgAAekUI60E9qUoihAEAgN4RwnowWQmjOR8AAPSGENaDtFkJ20clDAAA9IYQ1oM0oRIGAAD6QwjrQTOEjVIJAwAAPSKE9SCt0ZgPAAD6QwjrQZ3pSAAA0CdCWA8mKmFMRwIAgB4RwnpAYz4AAOgXIawHkyGMShgAAOgNIawHTEcCAIB+EcJ6QGM+AADoFyGsBxyiAgAA9IsQ1oPJg7VSCQMAAL0hhPUgqVgVUwkDAAC9I4T1wLbSpEoIAwAAPSOE9SitVbSH6UgAANAjQliP6lTCAABAHwhhPUprFRrzAQBAzwhhPUqTCpUwAADQM0JYj2jMBwAA/SCE9SirhDEdCQAAekMI61G9VuXckQAAoGeEsB7REwYAAPpBCOsRe0cCAIB+EMJ6RGM+AADoByGsRzTmAwCAfhDCekRPGAAA6AchrEfsHQkAAPpBCOtRczoyIgY9FAAAcBAihPUorVXVCGnfOCEMAAB0jxDWozTJXjqa8wEAQC8IYT2aDGH0hQEAgO4RwnqU1qqSCGEAAKA3hLAeTVTCOGo+AADoASGsR2mSVcJGOUwFAADoQWkhzPZJtlfb3mR7o+2rplnnPNtP2t6Q//xhWeMpWlqjMR8AAPQuKfGxxyRdExHrbS+UtM72HRGxqW29b0TERSWOoxQ05gMAgH6UVgmLiEciYn1+eZekzZKWlPV8B1pzOpIQBgAAenFAesJsL5O0QtKaaW5+ie1v2v4n2y+c4f5X2l5re+327dtLHGnn6jUa8wEAQO9KD2G2F0i6WdLVEbGz7eb1kp4VEWdK+gtJn53uMSLiuohYGRErR0ZGyh1wh6iEAQCAfpQawmzXlAWwmyLilvbbI2JnROzOL39BUs32cWWOqSjNnrBRKmEAAKAHZe4daUnXS9ocER+cYZ1n5OvJ9tn5eHaUNaYiTe4dSSUMAAB0r8y9I8+VdLmk+21vyJddK+lkSYqIj0m6RNLbbY9JelrSpRFxUJwRm+lIAADQj9JCWETcJclzrPMRSR8pawxlqnOcMAAA0AeOmN+jedXm3pFUwgAAQPcIYT2yrTSpaJRKGAAA6AEhrA9pUqESBgAAekII60Naq9KYDwAAekII60OaVGjMBwAAPSGE9eGoek0bt+7kgK0AAKBrhLA+XP1zz9G3f7xLf/DZB3SQHN4MAAAMCUJYH85/4TP0zpc/W/+wbos+uebhQQ8HAAAcRAhhfbr6556rlz1vRO+5baPueeixQQ8HAAAcJAhhfapUrA9fukJLjzlCb//kev3oydFBDwkAABwECGEFOPqImq771ZX6yd4xvf2mdewxCQAA5kQIK8hzT1io//W6M3Xvw0/ov922adDDAQAAQ44QVqBfOONE/ZfzTtWqux/Wqrtp1AcAADMjhBXsmvOfp5c+d0R/9LmNWv/w44MeDgAAGFKEsIJVK9afX3qWnnF0XW//5Dpt20WjPgAA2B8hrASL5s/Txy9/kXY+PaZ33LReezm/JAAAaEMIK8kLTjxKf3rJct3z0ON63+dp1AcAAFMlgx7AoeziM5+pB7Y+qeu+/l2d/syj9foXnzToIQEAgCFBJaxkv/Pzz9O5z16s37n5Pr3mI3fpr//le3p0955BDwsAAAyYD7YTT69cuTLWrl076GF0ZfeeMa1a87BuvXerNj2yU9WK9dLnHKfXrlii8097ho6YVx30EAEAQAlsr4uIldPeRgg7sL79o1367Iat+ty9W/XDJ0d15LyqLjj9RP3iiiV6yamLVa140EMEAAAFIYQNoUYjtOZ7j+mz927VF+5/RLv2jOmEo1JdfOYz9R9OPkanHr9Az1o8X2lClQwAgIMVIWzIje4b11c2b9Ot927Vnd/eprFGtk0qlk4+dr5OHVmgnxo5UqeOLNCpxy/QqSMLdOyR81X4JSkAAAw5SURBVAY8agAAMJfZQhh7Rw6Beq2qVy0/Ua9afqJ+sndM393+lL6zfbe+0/x3227d9eCj2tNyvLFj5td00rHzdfzCVCML6zp+Yarjj0p1wsK6jj8q1fEL6zpuwTwlVfa9AABgGBHChsz8eYlOX3K0Tl9y9JTl443QD594Wg/moew725/S1iee1pbHn9a9Dz+hHU/t3e+xbGnxkfN03IJUC9JE89NEC9KqjpyX6Mg00YK0+W9VR+aXj6hVlSYV1WtVpbWK6kk1u9xcllRUoW8NAIC+EcIOEtWKddKx83XSsfP1sucdv9/t+8YbenT3Hm3buUfbdu3Rtl2j+vHOPdq+a1SP7t6rp/aM6cmf7NXWx8f01J5xPbVnTLv3jqmX2eh51YpqVSupVpRUrKRqJZWKqhOXs+vNy/OSiuYlVc2rVpQm2c+85k+15XLL9Vo1u1xLKppX9cSyWv7ctlWxVbFkWbay6xVNLm9ZJ7vNqjavV7Lbqrac36cZLd3ymK3XpWyKuFrJnh8AgH4Qwg4RtWpFJx59hE48+oiO7xMRenrfuHbvmQxmo/vGtWesMee/e8caGm+E9o03/w2NNxra1wiNj4fGGqGxRkNj46G9Yw09+fQ+7R1raO/Y5P33juf/jjUm+uAOJtU81DkPZs3A17yc5bRmIJwMdlYWEJ0vr9gTj1WttP3k4TGZ8pj5/aWpj5df15TnaXneZpic5j4TIbTl8sTy5g0KNRpSKBQhhaRGfiGUvZ+aW9HK7qv8Ofd7DWYMsTO9D5pBu/XxPPGYzceXNPHFIqZcjim3NbdfZeJ3tKqVycuVltdhPEKNRmg8QuMNabzR0Hgj+93HmpcbMbGdqvm/zS8p7ddtaawRGs9/ssvZ/4Hx8ebzZD/N94atifdHJR93873RfN813xcTr1hzWf4FovX1aX89ImLK6yZp4jVufZ0qrdfz524+TiPf/s3HakSoEVMfe6b3b6XSfP95v/dt8/003ft34j2X/07Z87W8RyfGNPX9GS3vWU1Z1vnnUOuXv/bXpPX1qvTQEdK+zSaWt27fKcv3v29T87VoxOQ2n3x9JpdXWt5f1fz/Q7VSmfh/0fw8kqVG/v6f+L/RyF77RjQvz77NWz9jJj5bWsbTvH/knzmt15vbbr/fRS0LNfnebf2Mnvycnvz/fvLi+Xr+M47qcgsVhxB2GLOt+fMSzZ+XSAsHO5ZmoNs73tC+sea/ob3j49o7NvW2fY3Y74N+4j9py/XmsuaHQnPd8fz+443QeGR/QFv/KE18aKvlD1N+YeKx8z/Kjfz+jfwPdPN5xhuTH/7ThZfWx2+uP+WnbVkz7EbLGNX6B6blQ7X1j0v7h9TUP1pT/0i1fnBO+RDMH7c9SDX/ALeGyezjdPJ5G43J168x5Xfe/w9M03SL2/+4h5rbTVO2e+sf7WxsLX+SWm5r/Z2z7akp75HptAbj1qDc/EBvvrfGxhv5l5DJ7TeXpDL1MZuhe+L93fKeiNDEZQD9ueyck/W+154xsOcnhGEoZH98sv4zYNCaIT00WXHq57GagWwsD+y1SkWVipRUKhMVp37H2Vr1a28zaL2ttcIk7V+lnbiPNCX0Nb98TPki01JFaa3wTmkVqEytXDVCE5Wr1i8HzZA++QVhasWuNYg3l7W2DDSrtu2V3+mrxS1VtSnVtvY60iyvvTTja9L65awR03+xmO1xpZYvUhPXp1tr6vL2WN58jSr7Vcg9Ua1urZCONRoTXyibX16b792JL5fR/LzOq01uqTBVJv+/uGVMzS+jrV9CW7/oVdrfNy3jm9iunroNW6uFzYJa69Zr3xbNL7et26sRoUXza11sneIRwgCgjZ31Nxb1WLWqVcb3iyLHOR2+EwHl4vgFAAAAA0AIAwAAGABCGAAAwAAQwgAAAAaAEAYAADAAhDAAAIABIIQBAAAMACEMAABgAAhhAAAAA0AIAwAAGABCGAAAwACUFsJsn2R7te1NtjfavmqWdV9se8z2JWWNBwAAYJiUeQLvMUnXRMR62wslrbN9R0Rsal3JdlXSn0r6UoljAQAAGCqlVcIi4pGIWJ9f3iVps6Ql06z6W5JulrStrLEAAAAMmwPSE2Z7maQVkta0LV8i6Rcl/eUc97/S9lrba7dv317WMAEAAA6YMqcjJUm2FyirdF0dETvbbv6wpN+NiIbtGR8jIq6TdF3+eNttf7+s8bY4TtKjB+B50D22zXBj+wwvts1wY/sMr362zbNmusER0eNjzs12TdLtkr4YER+c5vbvSWqmr+Mk/UTSlRHx2dIG1SHbayNi5aDHgf2xbYYb22d4sW2GG9tneJW1bUqrhDkrbV0vafN0AUySIuKUlvVvkHT7MAQwAACAspU5HXmupMsl3W97Q77sWkknS1JEfKzE5wYAABhqpYWwiLhLk1ONnax/RVlj6dF1gx4AZsS2GW5sn+HFthlubJ/hVcq2KbUnDAAAANPjtEUAAAADQAgDAAAYAEJYG9sX2P627Qdt/96gx3O4s/0J29tsP9Cy7Fjbd9j+f/m/xwxyjIermc4Py/YZDrbrtu+2/c18+7wnX36K7TX5Z9ynbM8b9FgPV7artu+1fXt+nW0zJGw/ZPt+2xtsr82XFf7ZRghrkZ/H8qOSfkHSaZLeaPu0wY7qsHeDpAvalv2epK9ExHMkfSW/jgOveX7Y0ySdI+kd+f8Xts9w2CPp5RFxpqSzJF1g+xxl5+r9UEQ8W9Ljkt42wDEe7q5Sdkq/JrbNcHlZRJzVcnywwj/bCGFTnS3pwYj4bkTslfT3kl4z4DEd1iLi65Iea1v8Gkk35pdvlPTaAzooSJr1/LBsnyEQmd351Vr+E5JeLukz+XK2z4DYXirpVZL+Kr9usW2GXeGfbYSwqZZI+kHL9S2a/qTjGKwTIuKR/PKPJJ0wyMFgv/PDsn2GRD7dtUHSNkl3SPqOpCciYixfhc+4wfmwpN+R1MivLxbbZpiEpC/ZXmf7ynxZ4Z9tpZ87EihTRIRtjrMyQO3nh209DyzbZ7AiYlzSWbYXSbpV0vMHPCRIsn2RpG0Rsc72eYMeD6b1sxGx1fbxku6w/a3WG4v6bKMSNtVWSSe1XF+aL8Nw+bHtEyUp/3fbgMdz2MrPD3uzpJsi4pZ8MdtnyETEE5JWS3qJpEW2m1/A+YwbjHMlXWz7IWVtLy+X9Gdi2wyNiNia/7tN2ReYs1XCZxshbKp7JD0n30NlnqRLJd024DFhf7dJenN++c2SPjfAsRy2Zjk/LNtnCNgeyStgsn2EpFcq69tbLemSfDW2zwBExLsjYmlELFP2d+arEfEmsW2Ggu0jbS9sXpZ0vqQHVMJnG0fMb2P7QmVz9VVJn4iIPx7wkA5rtldJOk/ScZJ+LOmPJH1W0qeVnYf0+5JeHxHtzfsome2flfQNSfdrsq/lWmV9YWyfAbO9XFnzcFXZF+5PR8R7bf+UsurLsZLulXRZROwZ3EgPb/l05Lsi4iK2zXDIt8Ot+dVE0t9FxB/bXqyCP9sIYQAAAAPAdCQAAMAAEMIAAAAGgBAGAAAwAIQwAACAASCEAQAADAAhDAA6ZPs827cPehwADg2EMAAAgAEghAE45Ni+zPbdtjfY/nh+Iuvdtj9ke6Ptr9geydc9y/a/277P9q22j8mXP9v2l21/0/Z626fmD7/A9mdsf8v2TW49WSYAdIEQBuCQYvsFkt4g6dyIOEvSuKQ3STpS0tqIeKGkryk7+4Ik/Y2k342I5cqO/t9cfpOkj0bEmZJ+RtIj+fIVkq6WdJqkn1J2HkAA6Foy9yoAcFB5haQXSbonL1IdoexEuw1Jn8rX+aSkW2wfLWlRRHwtX36jpH/Izxu3JCJulaSIGJWk/PHujogt+fUNkpZJuqv8XwvAoYYQBuBQY0k3RsS7pyy0/6BtvV7P2dZ6Lr9x8TkKoEdMRwI41HxF0iW2j5ck28fafpayz7tL8nV+RdJdEfGkpMdt/8d8+eWSvhYRuyRtsf3a/DFS2/MP6G8B4JDHNzgAh5SI2GT79yV9yXZF0j5J75D0lKSz89u2Kesbk6Q3S/pYHrK+K+kt+fLLJX3c9nvzx3jdAfw1ABwGHNFrRR4ADh62d0fEgkGPAwCamI4EAAAYACphAAAAA0AlDAAAYAAIYQAAAANACAMAABgAQhgAAMAAEMIAAAAG4P8DK9K0mCEcaioAAAAASUVORK5CYII=\n"},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","source":["### Test the model."],"metadata":{"id":"lIWgzz3dqmNH"}},{"cell_type":"code","source":["mlp_pred = []\n","original_test_file_excluding_last_two_chars = [] \n","states = None\n","\n","\n","for i in original_test_file:\n","    \n","    original_test_file_excluding_last_two_chars.append(i[:-2])\n","\n","    # next_char = tf.constant([i[:-2]])\n","    # result = [next_char]\n","\n","    # for j in range(2):\n","\n","    #     next_char, states = one_step_reloaded.generate_one_step(next_char, states=states)\n","    #     result.append(next_char)   \n","\n","    # mlp_pred.append( tf.strings.join(result)[0].numpy().decode(\"utf-8\") ) "],"metadata":{"id":"1O6fU9kmqmNH"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["mlp_pred"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673901169874,"user_tz":-60,"elapsed":542,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"ba7e92bc-e622-4be9-91cc-66086b425a97","id":"U4NH9lcpqmNI"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['Biographice\\r',\n"," 'Wheelbarr n',\n"," 'Nightgore',\n"," 'Counterpasi',\n"," 'Christm h',\n"," 'Squirin',\n"," 'Specksnyd s',\n"," 'Forecaste\\r',\n"," 'Affidave\\r',\n"," 'Hyeng',\n"," 'Monstroo ',\n"," 'Town-be',\n"," 'Massack\\r',\n"," 'Funer, ',\n"," 'Battering-R m']"]},"metadata":{},"execution_count":104}]},{"cell_type":"code","source":["original_test_file_excluding_last_two_chars"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"FpkStu8WorBi","executionInfo":{"status":"ok","timestamp":1673907086218,"user_tz":-60,"elapsed":5,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"edd8467a-4ef3-4ab9-e452-29f8cb405981"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['Biographic',\n"," 'Wheelbarr',\n"," 'Nightgo',\n"," 'Counterpa',\n"," 'Christm',\n"," 'Squir',\n"," 'Specksnyd',\n"," 'Forecast',\n"," 'Affidav',\n"," 'Hye',\n"," 'Monstro',\n"," 'Town-',\n"," 'Massac',\n"," 'Funer',\n"," 'Battering-R']"]},"metadata":{},"execution_count":247}]},{"cell_type":"markdown","source":["# LSTM"],"metadata":{"id":"o0i7KiNYA1ZL"}},{"cell_type":"markdown","metadata":{"id":"qNgAt_M2A38I"},"source":["## Build The Model"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"AINWX6OnA38J"},"outputs":[],"source":["# Length of the vocabulary in StringLookup Layer\n","vocab_size = len(ids_from_chars.get_vocabulary())\n","\n","# The embedding dimension\n","embedding_dim = 256\n","\n","# Number of RNN units\n","rnn_units = 1024"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"IvM8cybjA38J"},"outputs":[],"source":["class MyModelLSTM2(tf.keras.Model):\n","  def __init__(self, vocab_size, embedding_dim, rnn_units):\n","    super().__init__(self)\n","    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n","    self.gru = tf.keras.layers.LSTM(rnn_units,\n","                                   return_sequences=True,\n","                                   return_state=True)\n","    self.dense = tf.keras.layers.Dense(vocab_size)\n","\n","  def call(self, inputs, states=None, return_state=False, training=False):\n","    x = inputs\n","    x = self.embedding(x, training=training)\n","    # if states is None:\n","    #   states = self.gru.get_initial_state(x)\n","    x, states, _ = self.gru(x, initial_state=states, training=training)\n","    x = self.dense(x, training=training)\n","\n","    if return_state:\n","      return x, states, _\n","    else:\n","      return x"]},{"cell_type":"code","source":["# from tensorflow.keras.models import Sequential\n","# from tensorflow.keras.layers import Embedding, Dense, LSTM\n","\n","# # Define the Keras model\n","# model = Sequential()\n","# model.add(Embedding(vocab_size, embedding_dim))\n","# model.add(LSTM(rnn_units))\n","# model.add(Dense(vocab_size))"],"metadata":{"id":"H9HC1M79A38K"},"execution_count":null,"outputs":[]},{"cell_type":"code","execution_count":null,"metadata":{"id":"tu5f7juUA38K"},"outputs":[],"source":["model = MyModelLSTM2(\n","    vocab_size=vocab_size,\n","    embedding_dim=embedding_dim,\n","    rnn_units=rnn_units)"]},{"cell_type":"markdown","metadata":{"id":"MmcNwvUQA38L"},"source":["## Try the model\n","\n","<!-- Now run the model to see that it behaves as expected.\n","\n","First check the shape of the output: -->"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673905801855,"user_tz":-60,"elapsed":2649,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"4c829139-eee0-46e8-9677-c7f520f0517b","id":"dFUhzT87A38L"},"outputs":[{"output_type":"stream","name":"stdout","text":["(64, 100, 96) # (batch_size, sequence_length, vocab_size)\n"]}],"source":["for input_example_batch, target_example_batch in dataset.take(1):\n","    example_batch_predictions = model(input_example_batch)\n","    print(example_batch_predictions.shape, \"# (batch_size, sequence_length, vocab_size)\")"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673905805810,"user_tz":-60,"elapsed":1050,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"f3e43adb-6973-4214-85ca-0243884d5b1c","id":"73EiqjnFA38L"},"outputs":[{"output_type":"stream","name":"stdout","text":["Model: \"my_model_lstm2_7\"\n","_________________________________________________________________\n"," Layer (type)                Output Shape              Param #   \n","=================================================================\n"," embedding_18 (Embedding)    multiple                  24576     \n","                                                                 \n"," lstm_14 (LSTM)              multiple                  5246976   \n","                                                                 \n"," dense_20 (Dense)            multiple                  98400     \n","                                                                 \n","=================================================================\n","Total params: 5,369,952\n","Trainable params: 5,369,952\n","Non-trainable params: 0\n","_________________________________________________________________\n"]}],"source":["model.summary()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"58GyS6-XA38M"},"outputs":[],"source":["sampled_indices = tf.random.categorical(example_batch_predictions[0], num_samples=1)\n","sampled_indices = tf.squeeze(sampled_indices, axis=-1).numpy()"]},{"cell_type":"markdown","metadata":{"id":"DFa-gK_tA38M"},"source":["This gives us, at each timestep, a prediction of the next character index:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673905813946,"user_tz":-60,"elapsed":7,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"7034ae87-eb92-4234-ad6d-b69d8168e7ed","id":"zOIbivsoA38M"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([25, 71, 61, 51, 22, 90,  3, 30, 85, 73, 66, 21, 52, 25, 31, 84, 35,\n","       78, 93, 81, 48,  1, 66, 77, 29,  8, 92, 65, 70, 77,  9, 63, 76, 32,\n","       73, 13, 59, 33, 38, 75,  1, 74, 95, 74, 14, 28, 41, 17, 67,  9, 42,\n","       31, 14, 59, 52, 21, 27, 45, 48, 14,  4, 74, 70,  6, 42, 44, 59, 25,\n","       67,  4, 38, 55, 46, 48, 13, 67, 77, 26, 21, 22, 84, 21, 37, 83,  5,\n","       25,  1, 44, 52, 15,  8, 51, 69, 59, 80, 21, 47, 54, 40, 10])"]},"metadata":{},"execution_count":219}],"source":["sampled_indices"]},{"cell_type":"markdown","metadata":{"id":"kaMhMU3GA38N"},"source":["Decode these to see the text predicted by this untrained model:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673905818005,"user_tz":-60,"elapsed":4,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"6bdd4e07-f619-463d-967e-946d2e2ae3d4","id":"pnoizw7IA38N"},"outputs":[{"output_type":"stream","name":"stdout","text":["Input:\n"," b'moving. The cabin entrance was locked within; the\\r\\nhatches were all on, and lumbered with coils of r'\n","\n","Next Char Predictions:\n"," b'8mcV5\\xc5\\x93 A\\xc2\\xa3oh4W8BzFt\\xe2\\x80\\x99wS\\nhs?&\\xe2\\x80\\x98gls\\'erCo,aDIq\\np\\xe2\\x80\\x9dp-;L0i\\'MB-aW4:PS-!pl$MOa8i!IZQS,is945z4Hy\"8\\nOW.&Vkav4RYK('\n"]}],"source":["print(\"Input:\\n\", text_from_ids(input_example_batch[0]).numpy())\n","print()\n","print(\"Next Char Predictions:\\n\", text_from_ids(sampled_indices).numpy())"]},{"cell_type":"markdown","metadata":{"id":"BKgUYxv4A38N"},"source":["## Train the model"]},{"cell_type":"markdown","metadata":{"id":"-kII77Q-A38N"},"source":["### Attach an optimizer, and a loss function"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iB3IBZshA38O"},"outputs":[],"source":["loss = tf.losses.SparseCategoricalCrossentropy(from_logits=True)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673905844754,"user_tz":-60,"elapsed":4,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"0279dba2-b453-4afb-e328-4e23ae783927","id":"tFM1nSHOA38O"},"outputs":[{"output_type":"stream","name":"stdout","text":["Prediction shape:  (64, 100, 96)  # (batch_size, sequence_length, vocab_size)\n","Mean loss:         tf.Tensor(4.564588, shape=(), dtype=float32)\n"]}],"source":["example_batch_mean_loss = loss(target_example_batch, example_batch_predictions)\n","print(\"Prediction shape: \", example_batch_predictions.shape, \" # (batch_size, sequence_length, vocab_size)\")\n","print(\"Mean loss:        \", example_batch_mean_loss)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673905849422,"user_tz":-60,"elapsed":5,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"9ffd9562-1133-46c2-b20d-5ed5ae9bb7b5","id":"OAlhfZEUA38P"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["96.02303"]},"metadata":{},"execution_count":223}],"source":["tf.exp(example_batch_mean_loss).numpy()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"c3bobTrYA38P"},"outputs":[],"source":["model.compile(optimizer='adam', loss=loss)"]},{"cell_type":"markdown","metadata":{"id":"OuG8pHWXA38Q"},"source":["### Configure checkpoints"]},{"cell_type":"markdown","metadata":{"id":"0l4bKxfJA38Q"},"source":["`tf.keras.callbacks.ModelCheckpoint` to ensure that checkpoints are saved during training:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"m8gs_xlPA38Q"},"outputs":[],"source":["# Directory where the checkpoints will be saved\n","checkpoint_dir = './training_checkpoints_lstm2'\n","# Name of the checkpoint files\n","checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt_{epoch}\")\n","\n","checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n","    filepath=checkpoint_prefix,\n","    save_weights_only=True)"]},{"cell_type":"markdown","metadata":{"id":"VaQtAxjHA38Q"},"source":["### Execute the training"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"sHsBXyhpA38R"},"outputs":[],"source":["EPOCHS = 50"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"outputId":"b0228a9e-87fb-4636-f056-57e7102352ab","id":"GV_qpN1dA38R","executionInfo":{"status":"ok","timestamp":1673906691197,"user_tz":-60,"elapsed":825028,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch 1/50\n","193/193 [==============================] - 15s 67ms/step - loss: 2.7023\n","Epoch 2/50\n","193/193 [==============================] - 14s 69ms/step - loss: 2.1192\n","Epoch 3/50\n","193/193 [==============================] - 15s 72ms/step - loss: 1.8806\n","Epoch 4/50\n","193/193 [==============================] - 15s 70ms/step - loss: 1.7273\n","Epoch 5/50\n","193/193 [==============================] - 14s 69ms/step - loss: 1.6223\n","Epoch 6/50\n","193/193 [==============================] - 14s 69ms/step - loss: 1.5484\n","Epoch 7/50\n","193/193 [==============================] - 14s 69ms/step - loss: 1.4895\n","Epoch 8/50\n","193/193 [==============================] - 14s 69ms/step - loss: 1.4410\n","Epoch 9/50\n","193/193 [==============================] - 15s 71ms/step - loss: 1.4014\n","Epoch 10/50\n","193/193 [==============================] - 15s 70ms/step - loss: 1.3683\n","Epoch 11/50\n","193/193 [==============================] - 14s 69ms/step - loss: 1.3384\n","Epoch 12/50\n","193/193 [==============================] - 14s 69ms/step - loss: 1.3111\n","Epoch 13/50\n","193/193 [==============================] - 14s 69ms/step - loss: 1.2848\n","Epoch 14/50\n","193/193 [==============================] - 15s 70ms/step - loss: 1.2599\n","Epoch 15/50\n","193/193 [==============================] - 15s 70ms/step - loss: 1.2357\n","Epoch 16/50\n","193/193 [==============================] - 15s 69ms/step - loss: 1.2114\n","Epoch 17/50\n","193/193 [==============================] - 16s 70ms/step - loss: 1.1868\n","Epoch 18/50\n","193/193 [==============================] - 15s 70ms/step - loss: 1.1614\n","Epoch 19/50\n","193/193 [==============================] - 15s 70ms/step - loss: 1.1359\n","Epoch 20/50\n","193/193 [==============================] - 14s 69ms/step - loss: 1.1088\n","Epoch 21/50\n","193/193 [==============================] - 15s 70ms/step - loss: 1.0810\n","Epoch 22/50\n","193/193 [==============================] - 15s 70ms/step - loss: 1.0515\n","Epoch 23/50\n","193/193 [==============================] - 14s 69ms/step - loss: 1.0208\n","Epoch 24/50\n","193/193 [==============================] - 15s 69ms/step - loss: 0.9896\n","Epoch 25/50\n","193/193 [==============================] - 15s 71ms/step - loss: 0.9552\n","Epoch 26/50\n","193/193 [==============================] - 15s 71ms/step - loss: 0.9211\n","Epoch 27/50\n","193/193 [==============================] - 14s 68ms/step - loss: 0.8867\n","Epoch 28/50\n","193/193 [==============================] - 15s 70ms/step - loss: 0.8509\n","Epoch 29/50\n","193/193 [==============================] - 15s 71ms/step - loss: 0.8144\n","Epoch 30/50\n","193/193 [==============================] - 14s 69ms/step - loss: 0.7794\n","Epoch 31/50\n","193/193 [==============================] - 14s 69ms/step - loss: 0.7447\n","Epoch 32/50\n","193/193 [==============================] - 14s 69ms/step - loss: 0.7093\n","Epoch 33/50\n","193/193 [==============================] - 15s 70ms/step - loss: 0.6743\n","Epoch 34/50\n","193/193 [==============================] - 14s 69ms/step - loss: 0.6417\n","Epoch 35/50\n","193/193 [==============================] - 15s 70ms/step - loss: 0.6111\n","Epoch 36/50\n","193/193 [==============================] - 14s 69ms/step - loss: 0.5807\n","Epoch 37/50\n","193/193 [==============================] - 15s 70ms/step - loss: 0.5534\n","Epoch 38/50\n","193/193 [==============================] - 15s 71ms/step - loss: 0.5269\n","Epoch 39/50\n","193/193 [==============================] - 15s 70ms/step - loss: 0.5006\n","Epoch 40/50\n","193/193 [==============================] - 15s 70ms/step - loss: 0.4768\n","Epoch 41/50\n","193/193 [==============================] - 14s 69ms/step - loss: 0.4564\n","Epoch 42/50\n","193/193 [==============================] - 14s 69ms/step - loss: 0.4323\n","Epoch 43/50\n","193/193 [==============================] - 15s 70ms/step - loss: 0.4149\n","Epoch 44/50\n","193/193 [==============================] - 15s 70ms/step - loss: 0.3984\n","Epoch 45/50\n","193/193 [==============================] - 14s 69ms/step - loss: 0.3827\n","Epoch 46/50\n","193/193 [==============================] - 14s 69ms/step - loss: 0.3648\n","Epoch 47/50\n","193/193 [==============================] - 14s 69ms/step - loss: 0.3501\n","Epoch 48/50\n","193/193 [==============================] - 15s 70ms/step - loss: 0.3375\n","Epoch 49/50\n","193/193 [==============================] - 15s 70ms/step - loss: 0.3247\n","Epoch 50/50\n","193/193 [==============================] - 15s 69ms/step - loss: 0.3124\n"]}],"source":["history_lstm = model.fit(dataset, epochs=EPOCHS, callbacks=[checkpoint_callback])"]},{"cell_type":"markdown","metadata":{"id":"COjHh5_LA38S"},"source":["## Generate text"]},{"cell_type":"markdown","metadata":{"id":"72PXP5GrA38T"},"source":["The following makes a single step prediction:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NgAuPOLZA38T"},"outputs":[],"source":["class OneStep(tf.keras.Model):\n","  def __init__(self, model, chars_from_ids, ids_from_chars, temperature=1.0):\n","    super().__init__()\n","    self.temperature = temperature\n","    self.model = model\n","    self.chars_from_ids = chars_from_ids\n","    self.ids_from_chars = ids_from_chars\n","\n","    # Create a mask to prevent \"[UNK]\" from being generated.\n","    skip_ids = self.ids_from_chars(['[UNK]'])[:, None]\n","    sparse_mask = tf.SparseTensor(\n","        # Put a -inf at each bad index.\n","        values=[-float('inf')]*len(skip_ids),\n","        indices=skip_ids,\n","        # Match the shape to the vocabulary\n","        dense_shape=[len(ids_from_chars.get_vocabulary())])\n","    self.prediction_mask = tf.sparse.to_dense(sparse_mask)\n","\n","  @tf.function\n","  def generate_one_step(self, inputs, states=None):\n","    # Convert strings to token IDs.\n","    input_chars = tf.strings.unicode_split(inputs, 'UTF-8')\n","    input_ids = self.ids_from_chars(input_chars).to_tensor()\n","\n","    # Run the model.\n","    # predicted_logits.shape is [batch, char, next_char_logits]\n","    predicted_logits, states = self.model(inputs=input_ids, states=states,\n","                                          return_state=True)\n","    # Only use the last prediction.\n","    predicted_logits = predicted_logits[:, -1, :]\n","    predicted_logits = predicted_logits/self.temperature\n","    # Apply the prediction mask: prevent \"[UNK]\" from being generated.\n","    predicted_logits = predicted_logits + self.prediction_mask\n","\n","    # Sample the output logits to generate token IDs.\n","    predicted_ids = tf.random.categorical(predicted_logits, num_samples=1)\n","    predicted_ids = tf.squeeze(predicted_ids, axis=-1)\n","\n","    # Convert from token ids to characters\n","    predicted_chars = self.chars_from_ids(predicted_ids)\n","\n","    # Return the characters and model state.\n","    return predicted_chars, states"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"9ZsV7hZZA38U"},"outputs":[],"source":["one_step_model = OneStep(model, chars_from_ids, ids_from_chars)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"2u2xs468A38V"},"outputs":[],"source":["start = time.time()\n","states = None\n","next_char = tf.constant(['Gabriel '])\n","result = [next_char]\n","\n","for n in range(100):\n","  next_char, states, _ = one_step_model.generate_one_step(next_char, states=states)\n","  result.append(next_char)\n","\n","result = tf.strings.join(result)\n","end = time.time()\n","# print(result[0].numpy().decode('utf-8'), '\\n\\n' + '_'*80)\n","# print('\\nRun time:', end - start)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"WbmWwEkkA38W"},"outputs":[],"source":["start = time.time()\n","states = None\n","next_char = tf.constant(['Gabriel ', 'Gabriel ', 'Gabriel ', 'Gabriel ', 'Gabriel '])\n","result = [next_char]\n","\n","for n in range(1000):\n","  next_char, states = one_step_model.generate_one_step(next_char, states=states)\n","  result.append(next_char)\n","\n","result = tf.strings.join(result)\n","end = time.time()\n","print(result, '\\n\\n' + '_'*80)\n","print('\\nRun time:', end - start)"]},{"cell_type":"markdown","metadata":{"id":"ubBwadY0A38W"},"source":["## Export the generator\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1673905559773,"user_tz":-60,"elapsed":7394,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"236c0647-d293-4b10-c501-e703bf712182","id":"m2DrqK8nA38W"},"outputs":[{"output_type":"stream","name":"stderr","text":["WARNING:tensorflow:Skipping full serialization of Keras layer <__main__.OneStep object at 0x7f9e220be2e0>, because it is not built.\n","WARNING:absl:Found untraced functions such as lstm_cell_12_layer_call_fn, lstm_cell_12_layer_call_and_return_conditional_losses while saving (showing 2 of 2). These functions will not be directly callable after loading.\n"]}],"source":["tf.saved_model.save(one_step_model, 'one_step_lstm2')\n","one_step_reloaded = tf.saved_model.load('one_step_lstm2')"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"R-1mek5KA38X"},"outputs":[],"source":["states = None\n","next_char = tf.constant(['Gabriel '])\n","result = [next_char]\n","\n","for n in range(100):\n","  next_char, states = one_step_reloaded.generate_one_step(next_char, states=states)\n","  result.append(next_char)\n","\n","print(tf.strings.join(result)[0].numpy().decode(\"utf-8\"))"]},{"cell_type":"code","source":["plt.figure(figsize = (10,7))\n","plt.plot(history_lstm.history['loss'], label = \"LSTM\")\n","plt.title('model loss')\n","plt.ylabel('loss')\n","plt.xlabel('epoch');"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":458},"executionInfo":{"status":"ok","timestamp":1673906867238,"user_tz":-60,"elapsed":1067,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"f8da6f71-5251-4a8a-f0c1-7a521b4aaabc","id":"DU9PXCORA38X"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 720x504 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAmEAAAG5CAYAAADGcOOUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXyV5Z338e/vnOwhkIQkZDlACDuyE1lFENzFpW6107q1U5e20/aZTp9pZ9qZ1+ydp/N06lRrtdWxWh9HVLSKWi2KKyKEHdlkC0lISICEhCXruZ4/zkGRAmY7uc9JPu/XKy/OfZ/7nPOLd3v4cl3X/bvNOScAAAD0LJ/XBQAAAPRFhDAAAAAPEMIAAAA8QAgDAADwACEMAADAA4QwAAAADxDCAPQJZvaYmf1zO4/da2YXd/V9AOBcCGEAAAAeIIQBAAB4gBAGIGqEpwG/b2YbzeyYmT1iZoPM7FUzazCzZWaWccrx15jZR2ZWZ2ZvmdnYU56bYmZrw697WlLSaZ+1yMzWh1+7wswmdrLmr5vZTjM7bGYvmll+eL+Z2X+aWbWZ1ZvZJjMbH37uSjPbEq6twsz+qlP/wQDENEIYgGhzg6RLJI2SdLWkVyX9jaRshb6zvi1JZjZK0lOSvht+7hVJL5lZgpklSHpB0hOSMiU9E35fhV87RdKjku6WNFDSQ5JeNLPEjhRqZgsk/ZukmyXlSSqV9D/hpy+VdGH49xgQPuZQ+LlHJN3tnEuTNF7Smx35XAC9AyEMQLT5hXPugHOuQtK7kj50zq1zzjVKel7SlPBxX5T0snPuj865Fkn/ISlZ0mxJMyXFS/q5c67FOfespNWnfMZdkh5yzn3onGtzzv1WUlP4dR3xZUmPOufWOueaJP1Q0iwzK5TUIilN0hhJ5pzb6pyrDL+uRdI4M+vvnKt1zq3t4OcC6AUIYQCizYFTHp84w3a/8ON8hUaeJEnOuaCkMkkF4ecqnHPulNeWnvJ4qKTvhaci68ysTtLg8Os64vQajio02lXgnHtT0v2SHpBUbWYPm1n/8KE3SLpSUqmZvW1mszr4uQB6AUIYgFi1X6EwJSm0BkuhIFUhqVJSQXjfSUNOeVwm6V+cc+mn/KQ4557qYg2pCk1vVkiSc+6/nHPTJI1TaFry++H9q51z10rKUWjadHEHPxdAL0AIAxCrFku6yswWmlm8pO8pNKW4QtIHklolfdvM4s3seknTT3ntryXdY2YzwgvoU83sKjNL62ANT0m608wmh9eT/atC06d7zez88PvHSzomqVFSMLxm7ctmNiA8jVovKdiF/w4AYhQhDEBMcs5tl/QVSb+QdFChRfxXO+eanXPNkq6XdIekwwqtH1tyymtLJH1doenCWkk7w8d2tIZlkn4s6TmFRt+GS7ol/HR/hcJerUJTlock/TT83K2S9ppZvaR7FFpbBqCPsc8umQAAAEBPYCQMAADAA4QwAAAADxDCAAAAPEAIAwAA8ECc1wV0VFZWlissLPS6DAAAgM+1Zs2ag8657DM9F3MhrLCwUCUlJV6XAQAA8LnMrPRszzEdCQAA4AFCGAAAgAcIYQAAAB4ghAEAAHiAEAYAAOABQhgAAIAHCGEAAAAeIIQBAAB4gBAGAADgAUIYAACABwhhAAAAHiCEAQAAeIAQBgAA4AFCGAAAgAcIYadpam3Ttqp61Te2eF0KAADoxQhhp9lW2aDLf/6uVu465HUpAACgFyOEnSaQkSxJqqg74XElAACgNyOEnSYzNUFJ8T5V1BLCAABA5BDCTmNmCmSkqJwQBgAAIogQdgYF6clMRwIAgIgihJ1BQUayymuPe10GAADoxQhhZxDISFbt8RYdb271uhQAANBLEcLOoCA9fIUk68IAAECEEMLOIJCRIkkszgcAABFDCDuDk73CylmcDwAAIoQQdgbZ/RKV4PexOB8AAEQMIewMfD5TfnoSa8IAAEDEEMLOoiCDXmEAACByCGFnEUinaz4AAIgcQthZFGQkq6ahSY0tbV6XAgAAeiFC2Fmc7BW2nylJAAAQAYSwszjZpoJ1YQAAIBIIYWdRkEHXfAAAEDmEsLPI7Z8kv89YnA8AACKCEHYWcX6fcvsnMR0JAAAighB2DgUZyXTNBwAAEUEIO4dARjJrwgAAQEQQws4hkJ6sqvpGtbQFvS4FAAD0MoSwcwhkpCjopKojjV6XAgAAehlC2DmcbFPBFZIAAKC7EcLO4WTXfBbnAwCA7kYIO4e89CSZ0TUfAAB0v4iFMDMbbGbLzWyLmX1kZt85wzHzzeyIma0P//xdpOrpjMQ4v3LSEpmOBAAA3S4ugu/dKul7zrm1ZpYmaY2Z/dE5t+W04951zi2KYB1dEshIoU0FAADodhEbCXPOVTrn1oYfN0jaKqkgUp8XKQXpyUxHAgCAbtcja8LMrFDSFEkfnuHpWWa2wcxeNbPzzvL6u8ysxMxKampqIljpnwpkJGt/3Qm1BV2Pfi4AAOjdIh7CzKyfpOckfdc5V3/a02slDXXOTZL0C0kvnOk9nHMPO+eKnXPF2dnZkS34NAUZyWoNOlU30CsMAAB0n4iGMDOLVyiAPemcW3L68865eufc0fDjVyTFm1lWJGvqqE/bVDAlCQAAuk8kr440SY9I2uqc+9lZjskNHyczmx6u51CkauqMQEaKJLE4HwAAdKtIXh05R9KtkjaZ2frwvr+RNESSnHO/knSjpHvNrFXSCUm3OOeiavHVyZEwFucDAIDuFLEQ5px7T5J9zjH3S7o/UjV0h+QEv7L6JdA1HwAAdCs65rdDQXoya8IAAEC3IoS1Q0FGMmvCAABAtyKEtUMgI0UVdScUZcvVAABADCOEtUNBerKaWoM6eLTZ61IAAEAvQQhrh0DGyV5hLM4HAADdgxDWDgUZtKkAAADdixDWDnTNBwAA3Y0Q1g5pSfEakBzPFZIAAKDbEMLaqSA9melIAADQbQhh7RTISGZhPgAA6DaEsHY62bCVXmEAAKA7EMLaqSA9Wcea21R3vMXrUgAAQC9ACGunQEaKJNpUAACA7kEIa6dPG7YSwgAAQNcRwtqJrvkAAKA7EcLaaUByvFIT/ExHAgCAbkEIayczU0FGMtORAACgWxDCOiCQkULXfAAA0C0IYR1A13wAANBdCGEdEMhI1pETLWpopFcYAADoGkJYBxSEr5BkNAwAAHQVIawDCtLDbSoOE8IAAEDXEMI6gK75AACguxDCOiCrX4IS43yEMAAA0GWEsA74tFcYXfMBAEDXEMI6qCA9mV5hAACgywhhHRSgaz4AAOgGhLAOCmSk6NCxZp1obvO6FAAAEMMIYR10sk1FRR3rwgAAQOcRwjooEG7YypQkAADoCkJYB9E1HwAAdAdCWAflpCUp3m+MhAEAgC4hhHWQ32fKG0CbCgAA0DWEsE4oSKdhKwAA6BpCWCcEMpJZEwYAALqEENYJBRnJqm5oUlMrvcIAAEDnEMI6IZCRIuekyrpGr0sBAAAxihDWCZ82bGVKEgAAdA4hrBM+bdjK4nwAANA5hLBOyB2QJJ+JNhUAAKDTCGGdEO/3Kbd/ksqZjgQAAJ1ECOukQEYKXfMBAECnEcI6qSCDrvkAAKDzCGGdVJCerKr6RrW2Bb0uBQAAxCBCWCcFMpLVFnSqqqdXGAAA6DhCWCcVhNtUMCUJAAA6gxDWSYGMFElicT4AAOgUQlgn5Q1IkkTXfAAA0DmEsE5KivcrOy1RZYfpmg8AADqOENYF4/P7a/Xew3LOeV0KAACIMYSwLlgwdpD2Hjqu3QePeV0KAACIMYSwLlgwJkeS9MbWAx5XAgAAYg0hrAsK0pM1Nq+/lm2t9roUAAAQYwhhXXTx2BytKa1V3fFmr0sBAAAxhBDWRQvG5Kgt6PT2jhqvSwEAADGEENZFkwLpyuqXyJQkAADoEEJYF/l8pgVjsvXW9mq1cDNvAADQToSwbrBw7CA1NLZq9d7DXpcCAABiBCGsG1wwIksJfp/eZEoSAAC0EyGsG6QmxmnW8IF6YxshDAAAtA8hrJtcPDZHew4e066ao16XAgAAYgAhrJssGDtIEt3zAQBA+xDCuklBerLG5KbpDdaFAQCAdohYCDOzwWa23My2mNlHZvadMxxjZvZfZrbTzDaa2dRI1dMTLh47SCWltTpyvMXrUgAAQJSL5EhYq6TvOefGSZop6ZtmNu60Y66QNDL8c5ekByNYT8QtHBvqnv/WDkbDAADAuUUshDnnKp1za8OPGyRtlVRw2mHXSnrchayUlG5meZGqKdJC3fMT6J4PAAA+V4+sCTOzQklTJH142lMFkspO2S7XnwY1mdldZlZiZiU1NdF7j0afz3TR6By9Tfd8AADwOSIewsysn6TnJH3XOVffmfdwzj3snCt2zhVnZ2d3b4HdbOHYQapvbFXJ3lqvSwEAAFEsoiHMzOIVCmBPOueWnOGQCkmDT9kOhPfFrLkjQ93zaVUBAADOJZJXR5qkRyRtdc797CyHvSjptvBVkjMlHXHOVUaqpp6QmhinmXTPBwAAnyOSI2FzJN0qaYGZrQ//XGlm95jZPeFjXpG0W9JOSb+W9I0I1tNjTnbP3033fAAAcBZxkXpj59x7kuxzjnGSvhmpGryyYEyO/u73H+mNrdUqyu7ndTkAACAK0TE/AgIZKRqTm6ZlrAsDAABnQQiLkIVjc+ieDwAAzooQFiELxw6iez4AADgrQliETA6ka2BqAjf0BgAAZ0QIixCfz3TRmBy9Rfd8AABwBoSwCLp4bA7d8wEAwBkRwiJo7shsJfh9enMbV0kCAIDPIoRF0Cfd81kXBgAATkMIi7CFY3K0m+75AADgNISwCFs4NkeSGA0DAACfQQiLsJPd899gXRgAADgFIawHXDx2kFbvrdX+uhNelwIAAKIEIawH3DJ9sCTpkff2eFwJAACIFoSwHhDISNE1k/L11Kp9qjve7HU5AAAgChDCesjd84p0vLlNj39Q6nUpAAAgChDCesiY3P5aMCZHj63YqxPNbV6XAwAAPEYI60H3zh+uw8eatbikzOtSAACAxwhhPej8wkxNG5qhh9/ZzU29AQDo4whhPezeecNVUXdCL2+s9LoUAADgIUJYD1swJkcjc/rpV2/vknPO63IAAIBHCGE9zOcz3T1vuLZVNeitHTVelwMAADxCCPPANZPylT8gSQ++tcvrUgAAgEcIYR5IiPPpa3OLtGrPYa0prfW6HAAA4AFCmEduOX+w0lPi9au3GQ0DAKAvIoR5JDUxTrfNKtQftxzQzuoGr8sBAAA9jBDmoTtmFyop3qdfvb3b61IAAEAPI4R5KDM1QbecP0S/X1+hyiMnvC4HAAD0IEKYx/587jAFnfTIu3u8LgUAAPQgQpjHAhkpumZSvv7fqn2qO97sdTkAAKCHEMKiwN3zinS8uU1PfFDqdSkAAKCHEMKiwJjc/lowJkf/vWKvTjS3eV0OAADoAYSwKHHPvOE6fKxZz6wp87oUAADQAwhhUeL8wgxNG5qhh9/Zrda2oNflAACACCOERQkz0z3zhqu89oSeLmE0DACA3o4QFkUWjsnRrKKB+teXt6rs8HGvywEAABFECIsiPp/ppzdNlJnp+89uUDDovC4JAABECCEsygQyUvR3i8Zp5e7DemzFXq/LAQAAEUIIi0I3FQe0YEyO/v0P27Sr5qjX5QAAgAgghEUhM9NPrp+g5AS//nLxBq6WBACgFyKERamc/kn6p2vHa0NZnR56Z7fX5QAAgG5GCItiV0/K16KJefr5sh3asr/e63IAAEA3IoRFuX+6drzSUxL0l4vXq6mVWxoBANBbEMKiXEZqgv79hgnaVtWg+5Z97HU5AACgmxDCYsCCMYP0xeLB+tXbu7R2X63X5QAAgG5ACIsRP1o0VnkDkvVXizfoRDPTkgAAxDpCWIxIS4rXT2+aqN0Hj+nf/7DN63IAAEAXEcJiyOzhWbpjdqEeW7FXK3Ye9LocAADQBYSwGPPXl49RUVaqvv/sRtU3tnhdDgAA6CRCWIxJTvDrP26epMojJ/Q3SzZxk28AAGIUISwGTR2Soe9fNkZLN1bqb1/YJOcIYgAAxJo4rwtA59w7f7iONrXogeW7lBjn199fPU5m5nVZAACgnQhhMeyvLh2tppagfvPeHiXG+fSDK8YQxAAAiBGEsBhmZvrbq8aqqTWoh97ZrcR4v/7yklFelwUAANqBEBbjzEz/cM15ampt03+98bES43z65kUjvC4LAAB8DkJYL+Dzmf7t+olqag3qp69tV2KcT38+t8jrsgAAwDkQwnoJv8/0f2+apObWoP755a1KjPfr1plDvS4LAACcBS0qepE4v0/33TJFF4/N0Y9f2KzFq8u8LgkAAJwFIayXSYjz6f4/m6q5I7P010s26vfrK7wuCQAAnAEhrBdKivfr4VuLNWNYpv5y8Qa9uqnS65IAAMBp2hXCzOw7ZtbfQh4xs7Vmdmmki0PnJSf49cjt52vy4HT9xVPr9MTKUq9LAgAAp2jvSNhXnXP1ki6VlCHpVkk/iVhV6BapiXF67M7zNXdkln78wmb96IVNamkLel0WAABQ+0PYyTbsV0p6wjn30Sn7EMXSkuL1m9vP193zivS7lft02yOrVHus2euyAADo89obwtaY2esKhbDXzCxNEkMqMcLvM/3wirH62c2TtGZfra554D3tONDgdVkAAPRp7Q1hX5P0A0nnO+eOS4qXdGfEqkJEXD81oKfvmqnGlqC+8MD7WrblgNclAQDQZ7U3hM2StN05V2dmX5H0I0lHzvUCM3vUzKrNbPNZnp9vZkfMbH345+86Vjo6Y8qQDL34rTkqyu6nrz9Rol++tVPOOa/LAgCgz2lvCHtQ0nEzmyTpe5J2SXr8c17zmKTLP+eYd51zk8M//9jOWtBFeQOS9cw9s7RoYr7+zx+267tPr1djS5vXZQEA0Ke0N4S1utBwybWS7nfOPSAp7VwvcM69I+lwF+tDhCTF+/Vft0zW9y8brd+v368vPvSBqo40el0WAAB9RntDWIOZ/VCh1hQvm5lPoXVhXTXLzDaY2atmdt7ZDjKzu8ysxMxKampquuFjIUlmpm9eNEIP3zpNO6uP6pr739OHuw95XRYAAH1Ce0PYFyU1KdQvrEpSQNJPu/jZayUNdc5NkvQLSS+c7UDn3MPOuWLnXHF2dnYXPxanu/S8XD33jdlKTvDrll+v1D8v3cL0JAAAEdauEBYOXk9KGmBmiyQ1Ouc+b03Y571nvXPuaPjxK5LizSyrK++JzhuT21+vfHuuvjxjiH7z3h5d9V/van1ZnddlAQDQa7X3tkU3S1ol6SZJN0v60Mxu7MoHm1mumVn48fRwLcyFeSg1MU7/fN0EPfG16Tre3KYbHlyh/3htu5pbaQkHAEB3i2vncX+rUI+wakkys2xJyyQ9e7YXmNlTkuZLyjKzckl/r/A6MufcryTdKOleM2uVdELSLY5eCVFh7shs/eG7F+qflm7R/ct36o1t1fq/N03SuPz+XpcGAECvYe3JPWa2yTk34ZRtn6QNp+7rKcXFxa6kpKSnP7bPWrblgH6wZJOOnGjWdxaO1D3zhivO396lhAAA9G1mtsY5V3ym59r7t+kfzOw1M7vDzO6Q9LKkV7qrQESvi8cN0h//14W67Lxc/cfrO3TDgyu0s/qo12UBABDz2jUSJklmdoOkOeHNd51zz0esqnNgJMw7Szfu149f2KzjzW367sWjdOecQiXF+70uCwCAqHWukbB2h7BoQQjzVnVDo370/Ga9vuWAAhnJ+t+Xj9HVE/MUvsYCAACcotPTkWbWYGb1Z/hpMLP6yJSLaJaTlqSHbyvWk38+Q/2T4vXtp9bpul+u0Ko93BwBAICOYCQMnRYMOj2/rkI/fW27quobdem4QfrBFWNUlN3P69IAAIgKTEciok40t+nR9/fol8t3qqk1qC/PGKJvLxypgf0SvS4NAABPEcLQI2oamnTfGzv01KoypcT7de9Fw/XVOcNYvA8A6LMIYehRO6uP6ievbtOyrQeUNyBJX59bpFumD1ZKQnt7AwMA0DsQwuCJlbsP6Wev79CqvYeVnhKv22YO1W2zC5XFNCUAoI8ghMFTa0pr9fA7u/T6lgNK8Pt0U3FAX59bpKEDU70uDQCAiCKEISrsqjmqX7+zW0vWVqg1GNQVE/J094VFmhhI97o0AAAighCGqFJd36hH39+rJ1eWqqGpVbOHD9Td84brwpFZNH0FAPQqhDBEpYbGFj21ap8eeW+PDtQ3aWROP906a6i+MKVAaUnxXpcHAECXEcIQ1Zpbg3pxw349/sFebSw/otQEv66fGtBts4Zq5KA0r8sDAKDTCGGIGevL6vT4B3u1dEOlmtuCmlU0ULfNGqpLxg1SnP+cd9kCACDqEMIQcw4dbdLTJWV6cuU+VdSdUG7/JP3ZjCG6Zfpg5aQleV0eAADtQghDzGoLOr25rVqPf7BX7358UPF+06Xn5eqLxYM1Z0SW/D4W8gMAote5QhgtzBHV/D7TJeMG6ZJxg7S75qh+t3Kflqwr18sbK5U/IEk3TAvoxmkBeo4BAGIOI2GIOU2tbVq2pVqLS8r07sc1CjppZlGmbpo2WFdMyOX2SACAqMF0JHqtyiMntGRthRaXlKn00HH1S4zTool5uql4sKYOSafvGADAU4Qw9HrOOa3eW6vFJWV6eWOlTrS0aXh2qm6YFtB1kwuUn57sdYkAgD6IEIY+5WhTq17ZWKln1pRp9d5amUmzhw/UDVMDuuy8XKUmMl0JAOgZhDD0WaWHjun5dRVasrZC+w4fV0qCX5ePz9UNUwOaWTSQqysBABFFCEOf55xTSWmtlqwt19INlWpoalXegCR9YUqBrp8a0Iicfl6XCADohQhhwCkaW9q0bOsBPbemXO98fFBtQafxBf11zaR8LZqYz/oxAEC3IYQBZ1Hd0KgX1+/Xixv2a2P5EUnS9MJMXT05X1eOz9XAfokeVwgAiGWEMKAd9hw8ppc2hALZzuqj8vtMF4zI0jWT8nXpeYOUlhTvdYkAgBhDCAM6wDmnrZUNemnjfr24fr8q6k4oMc6nBWNydM2kfF00JkdJ8X6vywQAxABCGNBJzjmt3Venlzbs19KNlTp4tEkpCX5dPHaQFk3M04WjsglkAICzIoQB3aC1LagP9xzW0o379YfNVao93qK0xDhdMm6QFk3K0wUjspUQ5/O6TABAFCGEAd2spS2oFbsOaemG/XrtoyrVN7aqf1KcLjsvV4sm5Wv28IGK9xPIAKCvI4QBEdTcGtR7O2u0dEOlXt9yQEebWpWREq/Lx+dq0cR8msICQB9GCAN6SGNLm97ZUaOlGyu1bOsBHW9uU1a/BF0xPk+LJubp/MJM+QhkANBnEMIAD5xobtPy7dVaunG/3txWrcaWoAb1T9SVE/K0aGK+pg5JlxmBDAB6M0IY4LFjTa1atvWAlm6s1Nvba9TcFlRBerKunJCrqyfla0LBAAIZAPRChDAgitQ3tuiPHx3Q0o379e7HB9UadCocmKKrJ+Xr6kn5GjUozesSAQDdhBAGRKm648167aMqvbShUit2HVTQSaMHpenqSaEpy8KsVK9LBAB0ASEMiAHVDY16dVOVXtqwXyWltZKkiYEBumZSvq6amKe8AdxYHABiDSEMiDEVdSf08sb9emlDpTZVfHpj8Wsm5+uqCXnKSE3wuEIAQHsQwoAYdvqNxeP9pnmjsnXt5AJdPHaQkhO4bRIARCtCGNALOOf00f56/X59hV7csF8H6puUmuDXZeNzdd3kAs0ePlBxdOkHgKhCCAN6mbag04e7D+mF9RV6dVOVGppaldUvUVdPytN1kws0MUDLCwCIBoQwoBdrbGnT8m3VemF9hZZvC/UgK8pK1TWT83Xd5AKusAQADxHCgD7iyPEWvbq5Ui+sr9CHew7LOWny4HRdNzlfiyblK6tfotclAkCfQggD+qD9dSf00ob9en5dhbZVNcjvM10wIkvXTcnXpeNylZoY53WJANDrEcKAPm57VYNeWF+hF9fvV0XdCSXH+3XJuEG6fmqB5o7Mlp+bigNARBDCAEiSgkGnktJavbC+Qi9vrNSREy3K7Z+k66cW6KbiwRrG+jEA6FaEMAB/oqm1TW9srdYzJWV6e0eNgk46vzBDN00brCsn5qkf05UA0GWEMADndKC+UUvWVuiZkjLtPnhMKQl+XTE+TzcXBzR9WCbtLgCgkwhhANrFOae1+2r1TEm5lm6s1NGmVg0dmKIbpwZ0Y3GA+1cCQAcRwgB02PHmVv1hc5WeKSnXB7sPyWfSvFHZ+uL5g7VgzCAlxNGdHwA+DyEMQJfsO3Rci0vK9MyaMh2ob9LA1ATdMC2gm4sHa0ROP6/LA4CoRQgD0C1a24J65+MaPb26TG9srVZr0Kl4aIZuPn+wrpqQR+8xADgNIQxAt6tpaNKSteV6uqRMu2uOKTXBr6sn5evLM4ZqQmCA1+UBQFQghAGIGOdCvceeXl2mlzdW6kRLmyYNTtdXZgzR1ZPylRTv97pEAPAMIQxAj6hvbNGSNeV6YmWpdtUc04DkeN00LaAvzxxKI1gAfRIhDECPcs5p5e7D+t3KUr32UZVag05zR2bpKzOHauGYHMX5ubISQN9wrhDGKloA3c7MNGv4QM0aPlDV9Y36n9VlemrVPt39xBrlDUjSl6YP0S3TBysnLcnrUgHAM4yEAegRrW1BvbGtWr9bWap3Pz6oeL/pqgl5umPOME0enO51eQAQEYyEAfBcnN+ny87L1WXn5WrPwWN6/IO9eqakXC+s36/Jg9N155xCXTE+jyawAPoMRsIAeOZoU6ueW1Oux1bs1Z6Dx5SdlqivzBiqP5sxRNlpiV6XBwBdxsJ8AFEtGHR6++MaPfb+Xr29o0YJfp8WTczTHXMKNTHAVCWA2MV0JICo5vOZLhqdo4tG52hXzVE98UGpnikp05J1FZo6JF13zhmmy8fnKp6rKgH0IhEbCTOzRyUtklTtnBt/hudN0n2SrpR0XNIdzrm1n/e+jIQBfUNDY4ueDU9Vlh46rrwBSbp11lB96fwhykhN8Lo8AGgXT6YjzexCSUclPX6WEHalpL9QKITNkHSfc3UelCYAABewSURBVG7G570vIQzoW9qCTsu3Veu/V+zR+zsPKSnepy9MCejOOYUaNSjN6/IA4Jw8mY50zr1jZoXnOORahQKak7TSzNLNLM85VxmpmgDEHr/PdPG4Qbp43CBtq6rXY+/v1ZK15Xpq1T7NHZmlr84ZpnmjsuXzmdelAkCHeLnAokBS2Snb5eF9f8LM7jKzEjMrqamp6ZHiAESfMbn99ZMbJuqDHy7U9y8brR0HGnTnY6t18c/e1m9X7NXx5lavSwSAdouJVa7OuYedc8XOueLs7GyvywHgsczUBH3zohF6768X6L5bJqt/crz+/sWPdMG/L9f9b36s+sYWr0sEgM/l5dWRFZIGn7IdCO8DgHaJ9/t07eQCXTu5QCV7D+uB5Tv1H6/v0EPv7Nbtswr11QuGKZNF/ACilJcjYS9Kus1CZko6wnowAJ1VXJip/75zupb+xQWaOzJLD7y1U3N+8qb+aekWHahv9Lo8APgTkbw68ilJ8yVlSTog6e8lxUuSc+5X4RYV90u6XKEWFXc65z73skeujgTQHjurG/TL5bv0+w375TfTjcUB3TtvuAZnpnhdGoA+hI75APqsfYeO61fv7NKzJeVqc07XTs7XN+aP0Iicfl6XBqAPIIQB6POqjjTq1+/u1pMflqqpNajLz8vVN+aP0ITAAK9LA9CLEcIAIOzQ0SY9tmKvHluxVw2NrbpwVLa+ddEITR+W6XVpAHohQhgAnKa+sUW/W1mqR97do0PHmnV+YYa+cdEIzR+VrdCSVQDoOkIYAJzFieY2LS4p00Nv79L+I40al9df37xohC4fnys/XfgBdBEhDAA+R3NrUL9fX6EH39ql3QePqSgrVffMH64vTClQvD8m+loDiEKEMABop7ag02sfVemB5Tv10f56FaQn6+55Rbq5eLCS4v1elwcgxhDCAKCDnHN6a3uN7l++U2tKa5Wdlqivzx2mL88YqtREL282AiCWEMIAoJOcc1q5O3RLpPd2HlR6SrzunD1Md8wu1ICUeK/LAxDlCGEA0A3W7avVA8t3adnWA+qXGKdbZw3V1y4Ypqx+iV6XBiBKEcIAoBttrazXA8t36uVNlUqM8+mW84fonnnDlTsgyevSAEQZQhgARMDumqN68K1den5dhXxm+tL0wbp3/gjCGIBPEMIAIILKDh/XL9/aqWdKygljAD6DEAYAPYAwBuB0hDAA6EGEMQAnEcIAwAOEMQCEMADw0GfCmM/05RlDdO/84cpJI4wBvR0hDACiQNnh47r/zZ16dm254v2m22YV6u4LizSQPmNAr0UIA4AoUnromO5742O9sK5CSfF+3TG7UF+fW6SM1ASvSwPQzQhhABCFdtUc1X3LPtZLG/crNSFOX51TqK/NLdKAZG6HBPQWhDAAiGLbqxp03xs79MqmKqUlxenrc4t055xCpSURxoBYRwgDgBjw0f4j+vmyj/XHLQeUnhKvr80ZptvnFKo/YQyIWYQwAIghG8vr9PNlH+vNbdVKS4rTnXOG6atzCpWewpoxINYQwgAgBm2uOKJfvPmxXvvogPolxum2WUP1tQuGcTUlEEMIYQAQw7ZV1esXb+7UK5sqlRTn11dmDtHXLyyizxgQAwhhANAL7Kxu0APLd+n36ysU7/fpS9OH6J55w+nAD0QxQhgA9CJ7Dh7TL5fv1PPrKuQz003FAd07f7gCGSlelwbgNIQwAOiFyg4f14Nv79IzJWVyTvrClAJ946IRGpaV6nVpAMIIYQDQi1UeOaGH3t6tp1btU0tbUFdPytc3LxqhUYPSvC4N6PMIYQDQB1Q3NOqRd/foiZWlOt7cpivG5+qbF43Q+IIBXpcG9FmEMADoQ2qPNevR9/fosff3qqGpVQvH5OhbC0ZoypAMr0sD+hxCGAD0QUdOtOjxFXv1yPt7VHe8RReMyNJ3Lh6p8wszvS4N6DMIYQDQhx1ratXvVpbq1+/u1sGjzZo9fKC+vXCkZhYN9Lo0oNcjhAEAdKK5TU9+WKqH3tmtmoYmzRiWqe9cPFKzigbKzLwuD+iVCGEAgE80trTpqVX79OBbu1Td0KTphZn69sKRmjOCMAZ0N0IYAOBPNLa06enVZXrwrV2qqm/UtKEZ+s7CkZo7MoswBnQTQhgA4KyaWtu0uKRcDy7fqf1HGjVlSLq+vWCk5o/OJowBXUQIAwB8rqbWNj23pkIPLN+piroTmlAwQN9aMEKXjB0kn48wBnQGIQwA0G4tbUE9v7ZCD7y1U6WHjmtMbpr+YsFIXTE+lzAGdBAhDADQYa1tQb20cb/uf3OndtUc04icfvrWRSO0aGKe4vw+r8sDYgIhDADQaW1Bp1c3V+oXb+zU9gMNKhyYom9cNEJfmFKgeMIYcE6EMABAlwWDTq9vOaBfvPmxPtpfr0BGsu6+sEg3Thus5AS/1+UBUYkQBgDoNs45Ld9erV+8uVPr9tUpMzVBt84cqttmDdXAfolelwdEFUIYAKDbOee0em+tHn5nl5ZtrVZinE83FQf05xcUqTAr1evygKhwrhAW19PFAAB6BzPT9GGZmj4sUzurG/Trd/Zo8epyPfnhPl02Lld3zSvS1CEZXpcJRC1GwgAA3aa6oVG/XbFXT3xQqvrGVp1fmKG7LhyuhWNyaG+BPonpSABAjzrW1KrFJWX6zbt7VFF3QkVZqbp9dqFumBZQv0QmYdB3EMIAAJ5obQvqlc1VevS9PVpfVqd+iXG6cVpAt88u1DDWjaEPIIQBADy3vqxOv12xV0s37ldLm9P80dm6fXah5o3MZqoSvRYhDAAQNaobGvXUh2X63Yelqmlo0rCsVN02a6hunBZQWlK81+UB3YoQBgCIOs2tQb26uVKPrdirdfvqlJrg143TArptdqGGZ/fzujygWxDCAABRbUN4qvKl8FTl3JFZun1WoS4akyM/U5WIYYQwAEBMqGlo0lOr9unJD0t1oL5JQzJTdOvMobq5eLAGpDBVidhDCAMAxJSWtqD+sLlKv12xVyWltUqO9+u6KQW6Y3ahRuemeV0e0G6EMABAzNpccUSPf7BXv1+/X02tQc0sytQdswt18dhBivP7vC4POCdCGAAg5tUea9b/rC7T71aWqqLuhHLSEnXDtIBumhZQEQv5EaUIYQCAXqO1Lag3tlVr8eoyLd9eraCTiodm6KbigK6amE9HfkQVQhgAoFeqrm/UknUVWlxSpt01x5Qc79eVE/J0c3FA04dlyowrK+EtQhgAoFdzzmntvjo9U1KmpRsrdbSpVUMHpuimaQHdMC2gvAHJXpeIPooQBgDoM443t+oPm6u0uKRMK3cflpl0wYgs3TA1oMvOy1Vygt/rEtGHEMIAAH3SvkPH9ezaci1ZW67y2hPqlxinKyfk6oapTFeiZxDCAAB9WjDotGrvYT23plyvbKrUseY2DclM0fVTC3TD1IAGZ6Z4XSJ6KUIYAABhJ6crn1tbrhW7Dsk5afqwTN04NaArJuRyE3F0K0IYAABnUFF3Qi+sq9Bza8q1++AxJcX7dNl5ubp+akAXjMjivpXoMs9CmJldLuk+SX5Jv3HO/eS05++Q9FNJFeFd9zvnfnOu9ySEAQC6m3NO68rqtGRtuV7aUKkjJ1qUk5ao66aEpiu5VRI6y5MQZmZ+STskXSKpXNJqSV9yzm055Zg7JBU7577V3vclhAEAIqmptU3Lt1Xr2TUVemt7tVqDTufl99f1UwO6ZlK+stMSvS4RMeRcISySbYWnS9rpnNsdLuJ/JF0racs5XwUAgIcS4/y6fHyeLh+fp0NHm/TShv1asq5C/7R0i/71la2aNypb107O18Kxg+jOjy6J5P96CiSVnbJdLmnGGY67wcwuVGjU7H8558pOP8DM7pJ0lyQNGTIkAqUCAPCnBvZL1B1zhumOOcP08YEGLVlXoRfWVejNbdVKjPNp3qhsXTUxj0CGTonkdOSNki53zv15ePtWSTNOnXo0s4GSjjrnmszsbklfdM4tONf7Mh0JAPBSMOi0Zl+tXt5YqVc3V+pAfZMSTgayCXlaODaHKyzxCa+mIyskDT5lO6BPF+BLkpxzh07Z/I2k/xPBegAA6DKfz3R+YabOL8zU3y0ap7X7arU0HMj+uOWAEuJ8unBktq6amKuLxw4ikOGsIhnCVksaaWbDFApft0j6s1MPMLM851xlePMaSVsjWA8AAN3K5zMVF2aq+JRA9vKmSr26qUrLth5QvD8U2OaPztb80TkamdOPLv34RKRbVFwp6ecKtah41Dn3L2b2j5JKnHMvmtm/KRS+WiUdlnSvc27bud6T6UgAQLQLBp3WldXq9Y8O6K3tNdp+oEGSlD8gSfNGZ2veqBzNGTGQUbI+gGatAAB4aH/dCb29o0Zvba/W+zsP6WhTq+J8puLCDM0fnaP5o7M1elAao2S9ECEMAIAo0dwa1JrSWr21o1pvb6/RtqrQKFnhwBRdNTFPV03I19g8AllvQQgDACBKVR1p1JvbqvXq5kqt2HVIbUGnouxULZqQp6sm5tOtP8YRwgAAiAGHjjbpDx9VaemGSn2455CCThqZ009XTczTool5GpFDIIs1hDAAAGJMdUOjXttcpZc2Vmr13sNyThqTm6ZLxg3SjGEDNXVoulISaBAb7QhhAADEsAP1jXplU6Ve3liptftqFXRSnM80vmCAZhRlasawTE0bmqkByVxtGW0IYQAA9BINjS1aU1qrVXsOa9Wew9pQXqeWNiczaWxuf00flqmZRaFmsgP7cbNxrxHCAADopU40t2ld2aehbO2+WjW2BCVJRVmpmjY0Q8WFGZo2NFPDs1O56rKHeXXbIgAAEGHJCX7NHp6l2cOzJIVaYGyqOKJVew5rTelhLdt6QM+sKZckZaTEa9rQUCArLszQhIIBSor3e1l+n0YIAwCgF0mI84WDVoak4XLOaVfNMa0pPaySvbVaU1qrZVurQ8f6fRpf0F/ThmZoypAMTRmSrrwByd7+An0I05EAAPQxh442aU1pKJCVlNZqU8URNbeGpjBz+ydpypD08A+jZV3FdCQAAPjEwH6JuvS8XF16Xq6k0BTm1sp6rdtXq3VldVq3r06vbq6SFLoKc1x+f00ZnK6pQzM0q2igcvoneVl+r8FIGAAA+BM1DU1aX1YXCmb76rShvE7Hm9skSUXZqZpZNDD8k6mcNELZ2XB1JAAA6JLWtqC2VjZo5e5D+mD3Ia3ac1hHm1olScPDoWzW8IGaMWygstNojXESIQwAAHSr1ragtlTW64Ndh7Ry9yGt3lv7mVA2eXCGJgYGaGJggMbm9e+z68oIYQAAIKJa24LavL9eK8OjZBvL63TwaLOk0Lqy0blp4VCWrgkFAzQ6N03xfp/HVUceIQwAAPQo55wqjzRqY/kRbSyv++TP+sbQaFlCnE/j8vprXH5/jclN0+hBaRqT218DUnrXrZcIYQAAwHPOOZUeOq6NFUe0sSwUzLZW1ashHMwkKW9Akkbnpml0blo4nPXX8JxUJcbF5nQmLSoAAIDnzEyFWakqzErVNZPyJX06Yra9qkHbqhq0vape26oa9P7Og2ppCw0U+X2moqxUjckLjZqNzQuNmuUNSIrp2zARwgAAgGfMTPnpycpPT9ZFY3I+2d/SFtSeg8c+CWbbqxq0trRWL23Y/8kx/ZPiNCavv8bmpn0S0EbnpiklITbiTWxUCQAA+pR4v0+jBqVp1KA0KTxqJkn1jS3aUdWgrVUN2lYZGjV7dk25joV7mJlJgzNSNGpQmkbn9gv/maairH5KiIuuCwEIYQAAIGb0T4pXcWGmigszP9kXDDpV1J3Q1sp6ba1s0I7qBu2oatDy7dVqC4amNON8pmFZqRoVvghg1KA0jS/or0BGile/CiEMAADENp/PNDgzRYMzUz65FZMkNbW2ac/BY9pe1aAdBxq0veqoNpUf0csbKyVJX5o+RP92/QSvyiaEAQCA3ikxzq8xuf01Jrf/Z/Yfa2rVzuqjSknw9opLQhgAAOhTUhPjNGlwutdlKLpWqAEAAPQRhDAAAAAPEMIAAAA8QAgDAADwACEMAADAA4QwAAAADxDCAAAAPEAIAwAA8AAhDAAAwAOEMAAAAA8QwgAAADxACAMAAPAAIQwAAMADhDAAAAAPEMIAAAA8YM45r2voEDOrkVTaAx+VJelgD3wOOo5zE904P9GLcxPdOD/RqyvnZqhzLvtMT8RcCOspZlbinCv2ug78Kc5NdOP8RC/OTXTj/ESvSJ0bpiMBAAA8QAgDAADwACHs7B72ugCcFecmunF+ohfnJrpxfqJXRM4Na8IAAAA8wEgYAACABwhhAAAAHiCEncbMLjez7Wa208x+4HU9fZ2ZPWpm1Wa2+ZR9mWb2RzP7OPxnhpc19lVmNtjMlpvZFjP7yMy+E97P+YkCZpZkZqvMbEP4/PxDeP8wM/sw/B33tJkleF1rX2VmfjNbZ2ZLw9ucmyhhZnvNbJOZrTezkvC+bv9uI4Sdwsz8kh6QdIWkcZK+ZGbjvK2qz3tM0uWn7fuBpDeccyMlvRHeRs9rlfQ959w4STMlfTP8/xfOT3RokrTAOTdJ0mRJl5vZTEn/Luk/nXMjJNVK+pqHNfZ135G09ZRtzk10ucg5N/mU/mDd/t1GCPus6ZJ2Oud2O+eaJf2PpGs9rqlPc869I+nwabuvlfTb8OPfSrquR4uCJMk5V+mcWxt+3KDQXyYF4vxEBRdyNLwZH/5xkhZIeja8n/PjETMLSLpK0m/C2ybOTbTr9u82QthnFUgqO2W7PLwP0WWQc64y/LhK0iAvi4FkZoWSpkj6UJyfqBGe7lovqVrSHyXtklTnnGsNH8J3nHd+Lul/SwqGtweKcxNNnKTXzWyNmd0V3tft321xXX0DwEvOOWdm9FnxkJn1k/ScpO865+pD/6AP4fx4yznXJmmymaVLel7SGI9LgiQzWySp2jm3xszme10PzugC51yFmeVI+qOZbTv1ye76bmMk7LMqJA0+ZTsQ3ofocsDM8iQp/Ge1x/X0WWYWr1AAe9I5tyS8m/MTZZxzdZKWS5olKd3MTv4DnO84b8yRdI2Z7VVo2csCSfeJcxM1nHMV4T+rFfoHzHRF4LuNEPZZqyWNDF+hkiDpFkkvelwT/tSLkm4PP75d0u89rKXPCq9heUTSVufcz055ivMTBcwsOzwCJjNLlnSJQuv2lku6MXwY58cDzrkfOucCzrlChf6eedM592VxbqKCmaWaWdrJx5IulbRZEfhuo2P+aczsSoXm6v2SHnXO/YvHJfVpZvaUpPmSsiQdkPT3kl6QtFjSEEmlkm52zp2+eB8RZmYXSHpX0iZ9uq7lbxRaF8b58ZiZTVRo8bBfoX9wL3bO/aOZFSk0+pIpaZ2krzjnmryrtG8LT0f+lXNuEecmOoTPw/PhzThJ/8859y9mNlDd/N1GCAMAAPAA05EAAAAeIIQBAAB4gBAGAADgAUIYAACABwhhAAAAHiCEAUA7mdl8M1vqdR0AegdCGAAAgAcIYQB6HTP7ipmtMrP1ZvZQ+EbWR83sP83sIzN7w8yyw8dONrOVZrbRzJ43s4zw/hFmtszMNpjZWjMbHn77fmb2rJltM7Mn7dSbZQJABxDCAPQqZjZW0hclzXHOTZbUJunLklIllTjnzpP0tkJ3X5CkxyX9tXNuokLd/0/uf1LSA865SZJmS6oM758i6buSxkkqUug+gADQYXGffwgAxJSFkqZJWh0epEpW6Ea7QUlPh4/5naQlZjZAUrpz7u3w/t9KeiZ837gC59zzkuSca5Sk8Putcs6Vh7fXSyqU9F7kfy0AvQ0hDEBvY5J+65z74Wd2mv34tOM6e8+2U+/l1ya+RwF0EtORAHqbNyTdaGY5kmRmmWY2VKHvuxvDx/yZpPecc0ck1ZrZ3PD+WyW97ZxrkFRuZteF3yPRzFJ69LcA0OvxLzgAvYpzbouZ/UjS62bmk9Qi6ZuSjkmaHn6uWqF1Y5J0u6RfhUPWbkl3hvffKukhM/vH8Hvc1IO/BoA+wJzr7Ig8AMQOMzvqnOvndR0AcBLTkQAAAB5gJAwAAMADjIQBAAB4gBAGAADgAUIYAACABwhhAAAAHiCEAQAAeOD/A7mED7Zqt0paAAAAAElFTkSuQmCC\n"},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","source":["### Test the model."],"metadata":{"id":"VoeK3qrUA38Y"}},{"cell_type":"code","source":["lstm_pred = []\n","original_test_file_excluding_last_two_chars = [] \n","states = None\n","\n","\n","for i in original_test_file:\n","    \n","    original_test_file_excluding_last_two_chars.append(i[:-2])\n","\n","    next_char = tf.constant([i[:-2]])\n","    result = [next_char]\n","\n","    for j in range(2):\n","\n","        next_char, states = one_step_reloaded.generate_one_step(next_char, states=states)\n","        result.append(next_char)   \n","\n","    lstm_pred.append( tf.strings.join(result)[0].numpy().decode(\"utf-8\") ) "],"metadata":{"id":"Lci2QjjhA38Y"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# lstm_pred"],"metadata":{"id":"2wjLpHsPA38Z"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## Aggregate Output"],"metadata":{"id":"9wgvF_8rYsj4"}},{"cell_type":"code","source":["plt.figure(figsize = (10,7))\n","plt.plot(history_lstm.history['loss'], label = \"LSTM\")\n","plt.plot(history.history['loss'], label = \"GRU\")\n","plt.plot(history_mlp.history['loss'], label = \"MLP\")\n","plt.title('model loss')\n","plt.legend()\n","plt.ylabel('loss')\n","plt.xlabel('epoch');"],"metadata":{"id":"geSv3lvKmkSj","colab":{"base_uri":"https://localhost:8080/","height":458},"executionInfo":{"status":"ok","timestamp":1673902948423,"user_tz":-60,"elapsed":1071,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"597d2b95-a2d1-4ac9-df5c-65661f44c74d"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 720x504 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAmEAAAG5CAYAAADGcOOUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXyU1d3+8c/JHpJAFrLAhCwo+xYQEFAxoLhQ617cLdpWbWuttrbaPl3U9mnVWp+fbW2tta1LK1arWOsOShBF9kWBgIAkJAGyQxKyZ87vjxliCCGZJDOZLNfbV15kZu77nm9CwItzzv09xlqLiIiIiPSsAH8XICIiIjIQKYSJiIiI+IFCmIiIiIgfKISJiIiI+IFCmIiIiIgfKISJiIiI+IFCmIgMCMaYp40xv/Tw2BxjzLndvY6ISHsUwkRERET8QCFMRERExA8UwkSk13BPA/7AGPOJMeaoMeavxphEY8xbxphKY8xyY0xMi+MvNsZsN8YcNsZkGWPGtXhtqjFmk/u8fwFhrd7rImPMFve5q40xk7tY8zeMMXuMMWXGmNeMMcPdzxtjzP8ZY4qMMRXGmE+NMRPdry00xuxw11ZgjLm7S98wEenTFMJEpLe5AlgAjAa+DLwF/BiIx/V31h0AxpjRwBLgTvdrbwL/NcaEGGNCgFeB54BY4CX3dXGfOxX4G3ArEAf8GXjNGBPamUKNMfOBXwOLgGFALvCC++XzgLnur2OI+5hS92t/BW611kYBE4H3O/O+ItI/KISJSG/ze2ttobW2AFgFrLXWbrbW1gJLganu464C3rDWLrPWNgCPAOHAHGAWEAz8P2ttg7X238D6Fu9xC/Bna+1aa22TtfYZoM59XmdcB/zNWrvJWlsH/AiYbYxJAxqAKGAsYKy12dbag+7zGoDxxpjB1tpya+2mTr6viPQDCmEi0tsUtvi8po3Hke7Ph+MaeQLAWusE8gCH+7UCa61tcW5ui89Tge+7pyIPG2MOAyPc53VG6xqqcI12Oay17wN/AB4HiowxTxpjBrsPvQJYCOQaY1YaY2Z38n1FpB9QCBORvuoArjAFuNZg4QpSBcBBwOF+7piUFp/nAf9rrY1u8THIWrukmzVE4JreLACw1v7OWnsaMB7XtOQP3M+vt9ZeAiTgmjZ9sZPvKyL9gEKYiPRVLwJfMsacY4wJBr6Pa0pxNfAx0AjcYYwJNsZcDsxsce5fgNuMMae7F9BHGGO+ZIyJ6mQNS4CbjDEZ7vVkv8I1fZpjjJnhvn4wcBSoBZzuNWvXGWOGuKdRKwBnN74PItJHKYSJSJ9krd0FXA/8HijBtYj/y9baemttPXA5sBgow7V+7JUW524AvoFrurAc2OM+trM1LAd+CryMa/TtFOBq98uDcYW9clxTlqXAb9yv3QDkGGMqgNtwrS0TkQHGHL9kQkRERER6gkbCRERERPxAIUxERETEDxTCRERERPxAIUxERETED4L8XUBnDR061Kalpfm7DBEREZEObdy4scRaG9/Wa30uhKWlpbFhwwZ/lyEiIiLSIWNM7sle03SkiIiIiB8ohImIiIj4gUKYiIiIiB/0uTVhIiIi0rs1NDSQn59PbW2tv0vpMWFhYSQnJxMcHOzxOQphIiIi4lX5+flERUWRlpaGMcbf5fictZbS0lLy8/NJT0/3+DxNR4qIiIhX1dbWEhcXNyACGIAxhri4uE6P/CmEiYiIiNcNlAB2TFe+XoUwERERET9QCBMREZF+JzIy8oTndu3aRWZmJhkZGYwbN45bbrmFd955h4yMDDIyMoiMjGTMmDFkZGRw4403kpWVhTGGp556qvkaW7ZswRjDI4880u0atTBfREREBoQ77riDu+66i0suuQSATz/9lEmTJnH++ecDkJmZySOPPML06dMByMrKYuLEibz44ot8/etfB2DJkiVMmTLFK/VoJExEREQGhIMHD5KcnNz8eNKkSR2ek5qaSm1tLYWFhVhrefvtt7nwwgu9Uo9GwkRERMRn7v/vdnYcqPDqNccPH8zPvzyh0+fdddddzJ8/nzlz5nDeeedx0003ER0d3eF5V155JS+99BJTp05l2rRphIaGdqXsE2gkTERERAaEm266iezsbL7yla+QlZXFrFmzqKur6/C8RYsW8dJLL7FkyRKuueYar9WjkTARERHxma6MWPnS8OHDufnmm7n55puZOHEi27Zt47TTTmv3nKSkJIKDg1m2bBmPPfYYq1ev9kotCmGt1DfVk1ORw7CIYUSFRPm7HBEREfGSt99+m3POOYfg4GAOHTpEaWkpDofDo3MfeOABioqKCAwM9Fo9CmGtfFb+Gde8cQ2PzXuM+Snz/V2OiIiIdEF1dfVxi/C/973vkZ+fz3e/+13CwsIA+M1vfkNSUpJH15szZ47Xa1QIa8UR6UrE+ZX5fq5EREREusrpdLb5/KOPPnrSc7Kyso57nJmZSWZm5gnH3Xfffd2o7AtamN9KdGg0EcERFFQV+LsUERER6ccUwloxxuCIdCiEiYiIiE8phLXBEenQdKSIiIj4lEJYG5Kjkjlw9ADWWn+XIiIiIv2UQlgbHJEOahprKK0t9XcpIiIi0k8phLUhOdJ1S6vWhYmIiIivKIS1QW0qRERE+rbCwkKuvfZaRo4cyWmnncbs2bNZunQpWVlZDBkyhIyMDMaOHcvdd9/dfM59993HI488ctx10tLSKCkp8UmNCmFtcES5QphGwkRERPoeay2XXnopc+fO5fPPP2fjxo288MIL5Oe7BlfOOusstmzZwubNm3n99df56KOP/FKnQlgbwoPCiQuLUwgTERHpg95//31CQkK47bbbmp9LTU3lO9/5znHHhYeHk5GRQUGBf/5/r475J+GIUpsKERGRbnvrXjj0qXevmTQJLnzwpC9v376dadOmdXiZ8vJydu/ezdy5c71Zncd8NhJmjAkzxqwzxmw1xmw3xtzfxjGhxph/GWP2GGPWGmPSfFVPZyVHJmskTEREpB/49re/zZQpU5gxYwYAq1atYsqUKTgcDs4///zm/SONMW2ef7Lnu8uXI2F1wHxrbZUxJhj40BjzlrV2TYtjvgaUW2tPNcZcDTwEXOXDmjzmiHTwTs47NDobCQrQgKGIiEiXtDNi5SsTJkzg5Zdfbn78+OOPU1JSwvTp0wHXmrDXX3+dffv2MWvWLBYtWkRGRgZxcXEcPHjwuGtVVlYSHR3tkzp9NhJmXarcD4PdH627n14CPOP+/N/AOcZXcbOTkqOSabJNHDp6yN+liIiISCfMnz+f2tpa/vSnPzU/V11dfcJx6enp3HvvvTz00EMAzJ07l9dee43KykoAXnnlFaZMmUJgYKBP6vTpEI8xJhDYCJwKPG6tXdvqEAeQB2CtbTTGHAHiAN/cC9oJx3qF5VflkxyV7OdqRERExFPGGF599VXuuusuHn74YeLj44mIiGgOWy3ddtttPPLII+Tk5DB58mRuv/12zjzzTIwxJCQk8NRTT/msTp+GMGttE5BhjIkGlhpjJlprt3X2OsaYW4BbAFJSUrxcZdua21RUFsCwHnlLERER8ZJhw4bxwgsvtPlaZmZm8+fh4eHH3R156623cuutt/q6PKCHWlRYaw8DK4ALWr1UAIwAMMYEAUOAE/YKstY+aa2dbq2dHh8f7+tyAUgclEigCdTifBEREfEJX94dGe8eAcMYEw4sAHa2Ouw14Kvuz68E3re9ZNfsoIAgkiKS1KZCREREfMKX05HDgGfc68ICgBetta8bYx4ANlhrXwP+CjxnjNkDlAFX+7CeTkuOUpsKERER8Q2fhTBr7SfA1Dae/1mLz2uBr/iqhu5KjkxmRd4Kf5chIiIi/ZC2LWqHI9JBWW0Z1Q0n3tYqIiIi0h0KYe041ppCU5IiIiLibQph7XBEuttUKISJiIj0KcYYrr/++ubHjY2NxMfHc9FFFwHw9NNPc/vtt59wXlpaGpMmTWLy5Mmcd955HDrku6btCmHtUAgTERHpmyIiIti2bRs1NTUALFu2DIfD4dG5K1as4JNPPmH69On86le/8lmNCmHtiA2LJTwoXG0qRERE+qCFCxfyxhtvALBkyRKuueaaTp0/d+5c9uzZ44vSAB93zO/rjDE4Ih3kVymEiYiIdMVD6x5iZ1nrNqHdMzZ2LPfMvKfD466++moeeOABLrroIj755BNuvvlmVq1a5fH7vP7660yaNKk7pbZLIawDyZHqFSYiItIXTZ48mZycHJYsWcLChQs9Pm/evHkEBgYyefJkfvnLX/qsPoWwDjiiHKw7tA5rLcYYf5cjIiLSp3gyYuVLF198MXfffTdZWVmUlp6wM2KbVqxYwdChQ31cmUJYh5Ijk6lurKa8rpzYsFh/lyMiIiKdcPPNNxMdHc2kSZPIysrydznH0cL8DjTfIVmpKUkREZG+Jjk5mTvuuKPN155++mmSk5ObP/Lze3YNuEbCOuCI+qJNxaR43y3OExEREe+pqqo64bnMzEwyMzMBWLx4MYsXLz7hmJycHN8W1oJGwjqQHOnqmq87JEVERMSbFMI6MCh4ELFhseoVJiIiIl6lEOYBR6RDbSpEREQ6wVrr7xJ6VFe+XoUwDzgiHRoJExER8VBYWBilpaUDJohZayktLSUsLKxT52lhvgeSo5JZnrucJmcTgQGB/i5HRESkVzt2p2FxcbG/S+kxYWFhJCcnd+ochTAPOCIdNNpGCqsLGR453N/liIiI9GrBwcGkp6f7u4xeT9ORHmjuFaZ1YSIiIuIlCmEeSI5yt6nQujARERHxEoUwDyRFJBFgAtQrTERERLxGIcwDwQHBJA1K0nSkiIiIeI1CmIccUWpTISIiIt6jEOah5MhkjYSJiIiI1yiEecgR6aCkpoTaxlp/lyIiIiL9gEKYhxxRrjYVB6oO+LkSERER6Q8UwjyUHOluU6E7JEVERMQLFMI8pF5hIiIi4k0KYR6KC4sjLDBMi/NFRETEKxTCWqmsbSBrVxElVXXHPW+MYXjkcI2EiYiIiFcohLWSW1rN4r+vZ/2+shNeS45SmwoRERHxDoWwVlLjBgGQU1p9wmuOSAcFVQVYa3u6LBEREelnFMJaiQoLZmhkCDklR094zRHpoKqhior6Cj9UJiIiIv2JQlgb0uIiyCk9MYTpDkkRERHxFoWwNqTGRZDbxnSkeoWJiIiItyiEtSF96CAOVdRSU9903POOSFfXfC3OFxERke5SCGtDalwEALllx09JRoZEEh0arelIERER6TaFsDakuUPYyRbnayRMREREukshrA2pQztuUyEiIiLSHQphbRgcFkxcRAi5bdwh6YhyhbAmZ1MbZ4qIiIh4RiHsJNKGRrCvjenI5MhkGp2NFNcU+6EqERER6S8Uwk4iNW5Q+20qtDhfREREukEhrLX6ashbx5jBjRw80kabiii1qRAREZHuUwhrrWQX/HUBU+12APaXHT8aNjxiOAajhq0iIiLSLQphrcWkAZBsCwFOWBcWHBhMYkQiBZUaCRMREZGuUwhrLTwGwoYQ13AAoO07JNWmQkRERLpJIawtMemEVuwnNiLkpL3CtDBfREREukMhrC0xaVC+j9S4QW12zU+OSqaopoi6prqer01ERET6BYWwtsSmw+H9jIwNa3M68libigNVB3q6MhEREeknFMLaEpMGzkYmRFZx4EgttQ2t2lREqk2FiIiIdI9CWFti0gEYE1oCnNimIjlKDVtFRESkexTC2uJuU5FCEcAJ68KGhg8lJCBEI2EiIiLSZQphbRmSDAFBxDceBCCn1bqwABPA8MjhCmEiIiLSZQphbQkIhOgUwipziRkU3GabiuSoZE1HioiISJf5LIQZY0YYY1YYY3YYY7YbY77bxjGZxpgjxpgt7o+f+aqeTotJh/IcUuMi2mxT4Yh0aOsiERER6bIgH167Efi+tXaTMSYK2GiMWWat3dHquFXW2ot8WEfXxKRBwUbS0yNYt6/shJeTI5OprK+kor6CwSGDe74+ERER6dN8NhJmrT1ord3k/rwSyAYcvno/r4tNh9rDjB7SyIEjNSe2qYhyt6nQHpIiIiLSBT2yJswYkwZMBda28fJsY8xWY8xbxpgJJzn/FmPMBmPMhuLiYh9W2oL7DsmxoWVYC3mt21S4G7ZqSlJERES6wuchzBgTCbwM3GmtrWj18iYg1Vo7Bfg98Gpb17DWPmmtnW6tnR4fH+/bgo9x9wpLD3C3qWi1OF8jYSIiItIdPg1hxphgXAHsn9baV1q/bq2tsNZWuT9/Ewg2xgz1ZU0ei0kFILHJ3aai1eL8wSGDiQqJYu2htdq+SERERDrNl3dHGuCvQLa19tGTHJPkPg5jzEx3PaW+qqlTQqMgIp7wqv1EDwo+oVcYwJmOM/mw4EPOf/l8rnztSh7f8jg7SndgrfVDwSIiItKX+PLuyDOAG4BPjTFb3M/9GEgBsNY+AVwJfNMY0wjUAFfb3pRgYtK+aFPRRgh7eO7DfDvj26zYv4IVeSt48pMneWLrEyRFJJGZnMm8lHnMSJxBcGBwz9cuIiIivZrpTZnHE9OnT7cbNmzomTd7+Ruwfw3fHfYsG3LK+eje+e0eXlZbxsq8lazIW8HHBz6mtqmWqOAoznScyezhs0kclEhseCxxYXHEhMUQFODLDCwiIiL+ZozZaK2d3tZrSgHtiUmDbf9m5PgQXttaQ11jE6FBgSc9PDYslstGXcZloy6jprGGNQfWsCJvBSvzV/JWzlsnHB8dGk1cWBxx4XHEhsU2/xoRHEFYYBihQaGEB4YTFhTW/NHycWhgKMEBwQQFBBFgtPmBiIhIX6IQ1p7YdLBOxg063Nym4tSEKI9ODQ8KZ17KPOalzKPJ2UReZR5ltWWU1pZSWlPq+rymlNJa1+c7SndQVltGVUNVl0oNMkEEB7oCWXBA8Bcfga5fA00gASbgi18DAk98zv1rgAnAGEOgCcQY43qOgC8+NwEYDMYYmv9zLe07/vkWvwbwxXVbnt/y2gbTfEzz5y2uc+y1Y+8DnPi41fPHHLtGe+e21voazc+3+npP9h4tzz/hPQwnvNbya21+3tD8PW5dU+trnqzek9bQzvWaP2/rkrb1w/ZH083xX2y3tP5+t/keXaitve9vy2tYazn2X8vHLc852Z+LY9c3GNcVbNvXOfZa66+zvZ+Djn7v29PebEjrP9ctv5bWdbR1nfa+/yf7PWt9XU+Ob/1enszwdPSz4am2/hy2/rnq1vXb+XuoI+19T9r6+lv/nLb8+6fl19X657b5WraTfx+cpN6WtR73NbRx7Y5+r0/29/Kx52LDYhkWOazda/iSQlh73L3CRgYWA8HklHgewloKDAgkbUgaaUPSOjy2rqmOmoYaaptqqW2sbf61prHmhMcNzgbXR1PDF5+3eFzvrG/+3GmdNNmm435tdDae8FzrD4v94nNrceI87nFbfxibn7fg5IvjrHVfy/3cseuLiIj4w1VjruIns37it/dXCGuPO4QNdx4CRrS5ON/bQgNDCQ0M9fn79CbWWpps03HBrWXIOxb+rLUnjBgcO7/l45bXPfZ8e8e0+a+rk4TD445v8a++1u9x3PntjBy1Pqf1vwDb+no9vWZHX09nzrHWejTy1J72rufR+Sf713GL72FHo31tXe9k12p5zbZGNluPErX8eW1vhKv5mh6OmLW+Tus6O/MPmZN939v6vrX53m38zHt7NNXT0Y62fr/bHSluq5DuDlQ1f1s6/rvGmzqzlru9P2ctvydtjvi2HOlq8bUeGyU7do22/jx4Wnfr38eOfj47M0p6st+Plj/TwyL8NwoGCmHti0yCoDAGHc1jSPjIHglhA5ExhiCjH0URERlYtJq7PQEBEJ0K5TmkxQ0it1XXfBEREZGuUgjrSGx6c6+wfSUaCRMRERHvUAjrSEwalO0jLW4QBw672lSIiIiIdJdCWEdi0qHhKGOianFayCur8XdFIiIi0g8ohHXEfYfkKcElAORqcb6IiIh4gUJYR2LTAUi2hQBaFyYiIiJeoRDWkegUACKO5hEVFqQ7JEVERMQrFMI6EhwOUcMx5TmkD41QrzARERHxCoUwT8SkNbepUAgTERERb1AI80RsOpTvIz1uEAXlNdQ3Ov1dkYiIiPRxCmGeiEmDyoOkRwe62lSUa12YiIiIdI9CmCdiXHdIjg4tBdSmQkRERLpPIcwT7l5hIygCIKdEI2EiIiLSPQphnnD3CouqziMqNEiL80VERKTbFMI8MSgOQiIxh3NJGxpBjnqFiYiISDcphHnCGNe6sLJ9pMYNIkdd80VERKSbFMI8FZMK5TmkxUWQX16tNhUiIiLSLQphnnI3bE2LC8dpoeBwjb8rEhERkT5MIcxTsenQVMeo8CoATUmKiIhItyiEecrdpiI1sBhAd0iKiIhItyiEecrdsHVITR6RoUEaCRMREZFuUQjz1JARYAIw5bmuOyTVpkJERES6QSHMU0EhMCQZyve5e4VpJExERES6TiGsM5rvkBxEfnkNDU1qUyEiIiJdoxDWGe6GrWlxETQ5LQXlalMhIiIiXaMQ1hkxaVBdwsghFoB9mpIUERGRLlII6wz3Rt7pAa42Fbm6Q1JERES6SCGsM9y9wmLqCogICdQdkiIiItJlCmGd4e4VZspzSI3THZIiIiLSdQphnREeDWHRUJ5D+tAIcjUSJiIiIl2kENZZselQvo/UuEHklVXTqDYVIiIi0gUKYZ3V3CssgkanpeCw2lSIiIhI5ymEdVZMOhzeT1psKAD7dIekiIiIdIFCWGfFpIGzkZEhhwG0LkxERES6RCGss9y9wuIaDjAoJFAjYSIiItIlCmGd5e4VZspzGJMUxYbcMv/WIyIiIn2SQlhnDXZAQDCU53DR5OFsK6hgd2Glv6sSERGRPkYhrLMCAiE6Bcr2cfGU4QQGGF7ZXODvqkRERKSPUQjrCnebivioUM4aNZT/bC7A6bT+rkpERET6EIWwrnA3bAW4bKqDA0dqWbOv1M9FiYiISF+iENYVMWlQewRqyjlvfBKRoUEs3aQpSREREfGcQlhXuDfypmwf4SGBXDAxibe2HaKmvsm/dYmIiEifoRDWFe42FZTnAHD5VAdVdY0syy70W0kiIiLStyiEdUVzCHOtC5s1Mo5hQ8JYuinffzWJiIhIn6IQ1hWhkRAR3zwSFhBguHSqgw92l1BcWeff2kRERKRPUAjrqph0KNvX/PDyqQ6anJb/bj3gx6JERESkr1AI66qYNCjPbX44KjGKiY7BvLJZU5IiIiLSMYWwropNh4p8aKxvfuqyqcnaxkhEREQ84rMQZowZYYxZYYzZYYzZboz5bhvHGGPM74wxe4wxnxhjpvmqHq+LSQPrhCN5zU9pGyMRERHxlC9HwhqB71trxwOzgG8bY8a3OuZCYJT74xbgTz6sx7uO9Qor/2JdmLYxEhEREU/5LIRZaw9aaze5P68EsgFHq8MuAZ61LmuAaGPMMF/V5FXH2lS0WJwP2sZIREREPNMja8KMMWnAVGBtq5ccQF6Lx/mcGNQwxtxijNlgjNlQXFzsqzI7JyoJgsKa21Qco22MRERExBM+D2HGmEjgZeBOa21FV65hrX3SWjvdWjs9Pj7euwV2lTEQNwoObj3uaW1jJCIiIp7waQgzxgTjCmD/tNa+0sYhBcCIFo+T3c/1DaPPh9yPoOr40TltYyQiIiId8eXdkQb4K5BtrX30JIe9BtzovktyFnDEWnvQVzV53YRLXXdI7vzvcU9rGyMRERHpiC9Hws4AbgDmG2O2uD8WGmNuM8bc5j7mTeBzYA/wF+BbPqzH+xInQuwpsP3V454OCDBckqFtjEREROTkgnx1YWvth4Dp4BgLfNtXNficMa7RsA//D46WQMTQ5pcun+bgiZV7+e/WA9x8ZrofixQREZHeSB3zu2vCZa4pyezjpyRHJ0YxYfhglqpxq4iIiLRBIay7jk1J7nj1hJcun5bMpwVH2FOkbYxERETkeAph3XVsSnLfKjh6fIPW5m2M1DNMREREWlEI84bxl4JtOuEuyeZtjLYc0DZGIiIichyFMG9ImgSxI0+4SxJc2xgVHK5h7b4yPxQmIiIivZVCmDcY4xoN2/fBCVOSzdsYbVbPMBEREfmCQpi3TGh7SvLYNkZvfnqII9UNfipOREREehuFMG9Jmgwx6W1OSd58RjpH6xv548o9fihMREREeiOFMG9pvkvyxCnJ8cMHc/nUZP7+UQ4Fh2v8VKCIiIj0Jgph3tR8l+TrJ7z0vfNGA/Dbd3f1dFUiIiLSCymEedOwKa4pyTYatzqiw7npjDSWbi5g+4EjfihOREREehOFMG86NiX5+UqoPrElxbcyT2VIeDAPvrXTD8WJiIhIb6IQ5m3tTEkOCQ/m9nmnsmp3Cat2F/uhOBEREektFMK8bdgUiElr8y5JgBtmp5IcE86v39ypLvoiIiIDmEKYtzU3bm17SjI0KJAfnD+GHQcr+M9W7SkpIiIyUCmE+cKES8HZCDvfaPPlL08eziTHEB555zNqG5p6uDgRERHpDRTCfGFYBkSntnmXJEBAgOFHC8dScLiGZz/O6dHSREREpHdQCPOF5rsks9qckgSYc8pQMsfE84f393C4ur5n6xMRERG/UwjzlfHtT0kC3HvhWCrrGnl8hbYzEhERGWgUwnxl+FSITjnplCTA2KTBXDktmWdW55JXVt2DxYmIiIi/KYT5ijEw4bJ2pyTBtZ2RMfDoss96rjYRERHxO4UwXzo2JbnrzZMeMmxIOF87M52lmwvYVqDtjERERAYKj0KYMea7xpjBxuWvxphNxpjzfF1cn3dsSvIkjVuPuS3zFGIGBfPrt7KxVg1cRUREBgJPR8JuttZWAOcBMcANwIM+q6q/ONa49fMsqCk/6WGDw4K545xRfLSnlA92l/RcfSIiIuI3noYw4/51IfCctXZ7i+ekPRMuBWcD7Dz5lCTAdaenkhI7iF+/mU2TtjMSERHp9zwNYRuNMe/iCmHvGGOiAKfvyupHhk/r8C5JgJCgAH5w/hh2HqrkmdU5PVObiIiI+I2nIexrwL3ADGttNRAM3OSzqvqTY1OSe9+Hw3ntHnrR5GGcOy6BX72ZzSn9FWEAACAASURBVIack99RKSIiIn2fpyFsNrDLWnvYGHM98BNAt/J5auYtYAJg5UPtHmaM4beLMkiOCedb/9xEUWVtDxUoIiIiPc3TEPYnoNoYMwX4PrAXeNZnVfU30SNg+tdgy/NQsrvdQ4eEB/PEDadRUdvA7f/cTEOTZn1FRET6I09DWKN19U64BPiDtfZxIMp3ZfVDZ30fgsJgxa86PHRs0mAevHwy63LKePCtnT1QnIiIiPQ0T0NYpTHmR7haU7xhjAnAtS5MPBUZD7O/BdtfgYNbOzz80qkOFs9J468f7uO/Ww/0QIEiIiLSkzwNYVcBdbj6hR0CkoHf+Kyq/mr27RAWDe//0qPDf7xwHNNTY7jn5U/4rLDSx8WJiIhIT/IohLmD1z+BIcaYi4Baa63WhHVWeDSceSfsfhdyP+7w8JCgAB6/bhqDQoK47bmNVNQ29ECRIiIi0hM83bZoEbAO+AqwCFhrjLnSl4X1WzNvhchEeO8B8GCLosTBYTx+7VRyy6q5+8Wt2tZIRESkn/B0OvJ/cPUI+6q19kZgJvBT35XVj4UMgrk/gP2rYc97Hp1y+sg4frxwHO/uKORPK/f6uEARERHpCZ6GsABrbVGLx6WdOFdam/ZVVxf99+4Hp2ctKG4+I42LJg/jkXd28aH2lxQREenzPA1Sbxtj3jHGLDbGLAbeANrfDFFOLigEMn8Mhz6B7Nc8OsUYw0NXTObUhEjueGEzBYdrfFykiIiI+JKnC/N/ADwJTHZ/PGmtvceXhfV7kxdB/FhY8b/Q1OjRKRGhQTxx/WnUNzr55j82UtvQ5OMiRURExFc8nlK01r5srf2e+2OpL4saEAICYf5PoOQz+OQFj08bGR/JbxdN4ZP8I9z78ic0qqO+iIhIn9RuCDPGVBpjKtr4qDTGVPRUkf3W2Itg+FTIehAa6zw+7fwJSfzg/DG8uuUA3/znJo2IiYiI9EHthjBrbZS1dnAbH1HW2sE9VWS/ZQyc8zM4kgcbn+7Uqd+edyr3XzyB5dmF3Pi3deohJiIi0sfoDkd/GzkP0s6CD34D9Uc7depX56Tx/67KYFNuOVf/eQ3FlZ6PpomIiIh/KYT527HRsKPFsOZPnT79kgwHf108g30lR7nyidXsL632QZEiIiLibQphvcGImTD6Qvjod1BT3unTzx4dzz+/cTpHahq44onVZB/Ucj0REZHeTiGst5j/E6g74gpiXTAtJYaXbp1NoDEs+vPHrM8p83KBIiIi4k0KYb1F0kSYeCWsfQIqD3XpEqMSo/j3N2cTHxnK9U+t5b3sQi8XKSIiIt6iENabzPsxOBvh5a9DU9fudkyOGcRLt81mTFIUtzy3kZc35nu5SBEREfEGhbDeJO4UuPj3kLMK3vwBWNu1y0SG8vw3ZjFrZCzff2krf/ngc2wXryUiIiK+oRDW20y5Gs64Ezb+HdY92eXLRIYG8bfFM1g4KYn/fTObW5/bSGmVWliIiIj0FgphvdE5P4cxC+Hte2HPe12+TGhQIH+4Zhr/s3AcWbuKOf//reL9nVonJiIi0hsohPVGAQFw+ZMQPw5euglKdnfjUoZvzB3Ja985g6GRIdz89AZ+vPRTqus92zRcREREfEMhrLcKjYJrX4DAYHj+KqjuXsuJsUmD+c/tZ3Dr3JEsWbefhY+tYvP+zvckExEREe9QCOvNolPgqn/A4f3w0uIu3zF5TGhQID9aOI4l35hFQ5Plyic+5tFln9HQ5PROvSIiIuIxn4UwY8zfjDFFxphtJ3k90xhzxBizxf3xM1/V0qelzoYvPwb7VsLbP/LKJWeNjOOtO8/ikozh/O693Vzxp9XsLa7yyrVFRETEM74cCXsauKCDY1ZZazPcHw/4sJa+bep1MOc7sP4vsP4pr1xycFgwjy7K4I/XTWN/WTVf+t0qnvs4R60sREREeojPQpi19gNAe+d4y7n3w+gL4M0fwudZXrvswknDePfOuZyeHsdP/7OdK/60mk1aKyYiIuJz/l4TNtsYs9UY85YxZsLJDjLG3GKM2WCM2VBcXNyT9fUeAYFw+V9g6Gh48atQutdrl04YHMbTN83g4Ssnk1dew+V/XM3tz28ir6zaa+8hIiIixzO+nH4yxqQBr1trJ7bx2mDAaa2tMsYsBB6z1o7q6JrTp0+3GzZs8HqtfUZ5DvxlPoTHwteXQ3i0Vy9/tK6RP3/wOU9+sBenhZvPSOdb805hcFiwV99HRERkIDDGbLTWTm/rNb+NhFlrK6y1Ve7P3wSCjTFD/VVPnxGT5rpjsjzHK60rWosIDeJ7C0az4u5MLpo8jCdW7iXzN1k8tyaXRt1FKSIi4jV+C2HGmCRjjHF/PtNdS6m/6ulTUufAFX+BA5vgrwug7HOvv8WwIeE8uiiD/95+JqMSIvnpq9u44LFVrNhZpMX7IiIiXuDLFhVLgI+BMcaYfGPM14wxtxljbnMfciWwzRizFfgdcLXV/909N+EyuPE1qC6Fp86F/Wt98jaTkofwwi2zePKG02hyWm56ej03/HUd2wqO+OT9REREBgqfrgnzhQG/Jqy10r3wzyvhSAFc9gRMvNxnb1Xf6OQfa3J57L3dHKlp4MxTh/L1s9I5e3Q87kFNERERaaG9NWEKYf3B0VJ44VrIWwPn3gdn3Ak+DEVHahp4fu1+/v7RPooq6xiTGMXXzkrnkozhhAYF+ux9RURE+hqFsIGgoRb+8y3Y9jJM+yp86beufSd9qL7RyX+3HuAvqz5n56FK4qNCWTwnjetOTyF6UIhP31tERKQvUAgbKJxOWPFLWPVbOGU+fOUZCBvs87e11vLhnhKe/OBzVu0uITw4kEXTk7n5zHRS4yJ8/v4iIiK9lULYQLPpWXj9Lhg6Bq57EYYk99hbZx+s4KlV+3htawFNTst545O4blYKZ5wylIAArRsTEZGBRSFsINq7Al68EYIHwbUvwPCpPfr2hRW1PLM6h+fX7edwdQOO6HAWTR/BV6YnMzw6vEdrERER8ReFsIGqcAc8vwiOlsC5P4eZt0JAz7aGq2ts4t3thfxrfR4f7inBGDh7dDxXzxjB/LGJhAT5e+csERER31EIG8gqC+G178Dud2DELLjkcRh6ql9KySur5qUNeby4IZ9DFbXERYRwxWnJLJo+glMTIv1Sk4iIiC8phA101sLWF+Dte6CxDub/BGZ9y7UpuB80OS0ffFbMv9bnsTy7kEanZUZaDFeelsyFk4Zpn0oREek3FMLEpfKQa8H+rjcheYZrVCx+jF9LKq6s45VN+fxrQx6fFx8lNCiABeMTuXyag7NGxRMcqOlKERHpuxTC5AvWwqf/hrd+APXVMO9HMPs7EBjk57IsW/OPsHRTPq9tPUB5dQNxESF8ecpwrpiWzETHYHXlFxGRPkchTE5UVQRvfA+y/wvDp7lGxRLH+7sqwNUEduVnxSzdnM/yHUXUNzk5NSGSy6Y6uHSqA4furhQRkT5CIUzaZi1sXwpv3g21FXD2PXDmnT7vtN8ZR6obeOPTgyzdnM/6nHKMgZlpsVw4MYkLJg4jaUiYv0sUERE5KYUwad/RElcQ274U4sfChQ/DyLP9XdUJ9pdWs3RzAW98eoDPCqsAmJoSzYUTk7hw4jBGxA7yc4UiIiLHUwgTz+x8E96+Fw7nwoTL4Lxf9mi3/c7YU1TFO9sP8da2g2wrqABgwvDBzSNkankhIiK9gUKYeK6hBj76HXz4KJgAmHs3zL4dgkL9XdlJ5ZVV8/a2Q7y57SCb9x8GYFRCJBdOTGLB+CQt6hcREb9RCJPOK8+Fd34MO1+H2JFwwUMw+jx/V9Whg0dqeHd7IW9tO8i6fWU4LSQODuWccYmcOy6BOacMJSzYP/3RRERk4FEIk67bsxzeugdK98DoC+GCX0Nsur+r8kjZ0XpW7CxieXYhH3xWzNH6JsKDAzlz1FAWjEtk3tgE4qN67wifiIj0fQph0j2N9bDmj7DyYXA2uu6gPONOCOk7C+HrGptY83kZ72UXsnxHIQeO1GIMTEmOZsH4RM4dl8joxEhNW4qIiFcphIl3VByAd38K2/4Ngx0w739gytV+2/6oq6y1ZB+sdAWy7EK25h8BYERsOAvGJbFgfCIz0mIIUrd+ERHpJoUw8a6cj+Ddn8CBTZAwAc69D0YtgD46ilRUUcvy7CKW7TjER3tLqW90MiQ8mPljEzh3XCJnj4knMtS/OwqIiEjfpBAm3nes0et7D0D5Pkg7CxbcD47T/F1Ztxyta2TV7mLe3VHI+zuLOFzdQEhgALNPiePc8a7F/cOGqGO/iIh4RiFMfKexHjY+DSsfguoSV3+x+T+FuFP8XVm3NTY52ZhbzrIdhSzLLiS3tBqA0YmRzB0Vz9lj4pmRFqu7LUVE5KQUwsT3aitg9e/h4z9AUz1Mvxnm/hAi4/1dmVdYa9lTVMWKXUWs/KyY9fvKqW9yEhYcwOnpcZw9Op65o+M5JT5Ci/tFRKSZQpj0nMpDkPUgbHoWgsPhjO/CrG9CaJS/K/Oq6vpG1n5exsrPivngs2I+LzkKgCM6nLmjh3L26HjOOHUoUWG9Zx9OERHpeQph0vOKP4P37nc1ex0UB2d+D2Z8zRXM+qG8smo+2O0KZB/tKaWqrpHgQMOMtFjmj00gc0yCRslERAYghTDxn/wN8P4v4PMsiBrm2gZp6o0QFOLvynymocnJptxyVuwqZsXOInYVVgKQEjuI+WMTmDc2gdPTtZZMRGQgUAgT/9u3yhXG8tZCdAqcfS9MvgoC+3/rh/zy6uZAtnpvCbUNTsKDAznj1KHMGxvPvDEJDI/unyOEIiIDnUKY9A7WurZBev8XcHArxI2CeT+G8ZdCwMBojFrb0MTHe0t5f2cR7+8souBwDeC64zJzTAKZo+OZnhZLSNDA+H6IiPR3CmHSu1gL2f+FFf8LxTshcRLM/x8YfUGfbfjaFcfuuMzaVUzWZ0Ws21dGQ5MlIiSQOae6FvdnjoknOabvbA8lIiLHUwiT3snZBNtehhW/cjV8TZ4B5/wM0uf6uzK/OFrXyOq9pWTtKiJrV3HzKNmpCZFkjo4nc0wCM9JjCA3SWjIRkb5CIUx6t6YG2PJPyHoIKg/AyHlwzk/7fPf97rDWsrf4KFnuvmRrPy+jvsnJoBD3WrIxCWSOiddaMhGRXk4hTPqGhlpY/xSs+i3UlMG4L8O8n0DCWH9X5nfV9Y18vLeUFbuKWLHzi1GyMYlRZLoX95+WGkOwNh0XEelVFMKkb6mtgDV/hNV/gIajrrsoM38EMan+rqxXaNm9f8XOYtbnlNHotESFBnHW6KFkjklg3pgE4qNC/V2qiMiApxAmfdPRUvjwUVj3F7BOmH4TnHU3RCX6u7JepbK2gY/2lJC1q5gVu4oorKjDGJiSHM254xI4Z1wiY5Oi1ChWRMQPFMKkbztSAB88DJueg6BQOP02OPNOCBvi78p6HWstOw5W8F52Ee9lF7I1/wjg2k7pHHcgmzUyVov7RUR6iEKY9A+le113Um77N4THwtk/dG0UHqRpt5Mpqqjl/Z1FLM8u4sM9xdQ2OIkICeSsUfGcMy6B+WMTiIvU909ExFcUwqR/ObAFlv0M9q2E6FRXW4sJlw+Yhq9ddaxR7PLsQt7LLuJQRS0BBk5LjeG88UksGJ9I2tAIf5cpItKvKIRJ/2Mt7H0Plt0HhZ/CsAxY8ACMPNvflfUJ1lq2H6hg2Y5C3t1RSPbBCsDVuf9YIJvkGEJAgNaRiYh0h0KY9F9OJ3z6Irz3C6jIh1MXwIL7IXGCvyvrU/LKqlm2o5BlOwpZl1NGk9OSNDiMc8cnsGB8ErNHxmkrJRGRLlAIk/6voRbWPQmrHnG1uMi41rUv5ZBkf1fW55Qfref9nUUs21HIys+KqWloIjI0iLPHxHPe+EQyRycwZFCwv8sUEekTFMJk4Kguc7W1WPtnMAEw5ztwxp0QGunvyvqk2oYmPtpTwrIdhSzPLqKkqo6gAMPM9FjOHZfIgvGJjIjV3pYiIiejECYDz+H9sPx+152UUcPgnJ+7mr5q8X6XOZ2WLfmHXYFsRyG7i6oAGJsUxYLxiZw7TuvIRERaUwiTgStvHbx1DxzYBMOnwQUPQsrp/q6qX8gpOcrybNfC/g05ZTgtJA4O5UuThnPZVAcTHYPVIFZEBjyFMBnYji3eX34fVB6EiVfCufdB9Ag/F9Z/HFtH9vb2Q6zcVUx9k5NT4iO4NMPBpVMdmrIUkQFLIUwEoK4KPnoMVv8OMHDGHXDGdyFEvbG86Uh1A29uO8jSzQWs21cGwPTUGC6Z6uCiScOIiQjxc4UiIj1HIUykpcP7YdnPYfsrEDXcNSo26StaL+YD+eXVvLb1AEs3FbC7qIrgQMPZoxO4dOpwzh2XSFiwtk8Skf5NIUykLfvXwNv3woHNkDwDFv4Ghk/1d1X90rE9LV/dXMBrWw9QWFFHZGgQF05M4tKpDmaNjCNQC/pFpB9SCBM5GacTti5xrRc7WgzTbnTdSRkR5+/K+q0mp+XjvaW8uqWAt7cdoqqukcTBoVyS4eCSjOGMH6YF/SLSfyiEiXSk9ghkPQRrn3D1FJv3E9fm4IFB/q6sX6ttaGJ5diGvbj5A1q4iGp2W0YmRzYEsOUYL+kWkb1MIE/FU0U5464euzcETJ8KFD0PaGf6uakAoO1rPG58e5D+bC9iQWw7AzLRYLp3qYOGkJKIHaUG/iPQ9CmEinWEtZL8G7/wPHMlztbQ47xcweLi/Kxsw8sqq+c+WApZuLmBv8dHmBf2XZLgW9IeHaEG/iPQNCmEiXVFfDR/+n6utRUAQzL0bZn8bgkL9XdmAYa1l+4EK/rPliwX9ESGBnD8hiYszhnPmqUMJCtRdrSLSe/klhBlj/gZcBBRZaye28boBHgMWAtXAYmvtpo6uqxAmPa5sn2tUbNcbEHsKXPgQjFrg76oGnCanZe2+Ul7bcoA3Pz1IRW0jcREhXDR5GBdnOJiWEq0F/SLS6/grhM0FqoBnTxLCFgLfwRXCTgces9Z2uJ+MQpj4ze7l8PY9ULoHxnwJLvgVxKT5u6oBqa6xiZW7ivnPlgMszy6krtHJiNhwrpw2gkUzkhk2JNzfJYqIAH6cjjTGpAGvnySE/RnIstYucT/eBWRaaw+2d02FMPGrxjpY80dY+RuwTXDGnXDmnRCs/+n7S2VtA+9uL2Tp5gI+3FNCgIH5YxO4ZmYKmWMS1H9MRPyqvRDmz/vvHUBei8f57ufaDWEifhUUCmfeBZMWwbKfwsoHYevzro3BxywETYf1uKiwYK44LZkrTktmf2k1/9qwnxc35LM8ewPDhoSxaPoIrpoxguHRCsoi0rv4cyTsdeBBa+2H7sfvAfdYa08Y5jLG3ALcApCSknJabm6uz2oW6ZR9q+DNH0BxNpx6LlzwEAw91d9VDXgNTU7eyy5iybr9fLC7GANkjnGNjs0bE6/F/CLSYzQdKeJLTQ2w7i+Q9WtoqIE5t8NZd7uavorf5ZVV86/1eby4IY+iyjoSB4dy1fQRLJoxQs1gRcTnemsI+xJwO18szP+dtXZmR9dUCJNeq7LQtf3R1uddG4Of/78w4TJNUfYSDU1O3t/pGh1b+VkxAGePjufamSnMH5ug0TER8Ql/3R25BMgEhgKFwM+BYABr7RPuFhV/AC7A1aLipramIltTCJNeb/9aePP7cOhTSD/btTF4/Bh/VyUt5JdX8+L6PP61IY/CijoSokK5asYIFk0fwYhYjY6JiPeoWatIT3M2wYa/wfu/gPqjMOubcPY9EBrl78qkhcYmJyt2FfP82lyy3KNjc0fFc83MFM4Zl0CwRsdEpJsUwkT8paoY3rsPNv8DoobBeb+EiVdoirIXKjhc41o7tj6PQxW1xEeFsmh6MtfMTNHaMRHpMoUwEX/LWwdvfB8OfQJpZ7mmKBPG+bsqaUNjk5OsXcUsWbefFbuKAJg3JoHrZ6Uyd3S8+o6JSKcohIn0Bs4m2Ph3eO8XUF8Fp98GmfdqirIXKzhcw5K1+3lhfR4lVXWMiA3n2pmpLJqeTFyk9hAVkY4phIn0JkdL4L37YdOzEJkE5/0CJn1FU5S9WH2jk3d3HOIfa3JZ83kZIYEBLJyUxPWzUjktNUZ7VorISSmEifRG+RtcU5QHt8CIWbDwYRg2xd9VSQd2F1byz7X7eXljPpV1jYxNiuK6WalcNtVBZKg/NyERkd5IIUykt3I2uRbtv/cAVJfCaYth/k8hIs7flUkHqusbeW3LAf6xNpdtBRVEhgZx5WnJ3DA7lVPi1ahXRFwUwkR6u5rDsPIhWPtnV6f9eT+B6TdDoEZWejtrLVvyDvPcx7m8/slB6pucnDVqKDfOTmP+WG0gLjLQKYSJ9BVFO+GtH8K+lZAwHi58CNLn+rsq8VBJVR3/Wp/HP9bkcvBILY7ocK6flcpVM0YQGxHi7/JExA8UwkT6Emth5+vwzo/h8H4Yf6lr8X50ir8rEw81NjlZnl3I06tzXAv5gwK4eMpwvjo7jUnJQ/xdnoj0IIUwkb6ooQZW/x5WPQpYOPMumHMHhKhxaF+y61Alz36cw9LNBVTXNzE1JZrFc9K4cOIwQoLUkV+kv1MIE+nLDufBsp/C9qUwZAQseEAbg/dBFbUN/HtDPs9+nENOaTXxUaFcd3oK156eQkJUmL/LExEfUQgT6Q9yPoS37oXCTyFlDlz4oFpa9EFOp2Xl7mKe/iiHlZ8VExxo+NKkYXx1ThpTU2L8XZ6IeJlCmEh/4WxyNXl9/xdQXQbTboRzfgYRQ/1dmXTB58VVPPtxLv/emE9VXSNTRkSzeE4qCycNIzQo0N/liYgXKISJ9Dc1h2Hlw7DuzxAcAZn3wIxvQJDuwOuLquoaeXljPs98nMPnxUcZGhnCtTNTuG5WKomDNVUp0pcphIn0V8WfwTs/gj3LIW4UXPBrGLXA31VJFzmdllV7SnhmdQ4rdhURaAwXThrG4jmpTEvR9kgifZFCmEh/Zi3sfhfe/hGU7YVR57kW7yeM83dl0g05JUd5bk0uL67Po7KukUmOIXx1ThoXTR5GWLCmKkX6CoUwkYGgsd41PbnyYaivginXQOaPIHqEvyuTbjha18grmwt4ZnUOe4qqiI0I4ZqZI7h+VirDhoT7uzwR6YBCmMhAcrQUPnwU1j3pejzjG3DW97UfZR9nrWX13lL+/lEO7+0sJMAYLpiQxFfnpDEjTVOVIr2VQpjIQHQ4D7IehK3PQ0ikq9HrrG+69qaUPi2vrJrn1uTywrr9VNQ2MjYpihtmp3JphoOIUO03KtKbKISJDGRF2fD+L11bIUUkwNk/hGlf1Z2U/UBNfROvbing2Y9zyT5YQVRoEFeclsz1s1I4NSHK3+WJCAphIgKQtw6W3we5H0FMGsz7CUy8AgK0dU5fZ61l0/5ynvs4lzc/PUR9k5PZI+O4YXYqC8YnEhyo32MRf1EIExEXa13tLJbf7+q8nzAezvyeaxukQE1j9QclVXW8uCGPf67ZT8HhGhIHh3LNzBSumZminmMifqAQJiLHczph+yvwwW+geCfEpLs2CJ9yjaYp+4kmpyVrVxHPfpzLys+KCQwwnD8hkcVz0rWQX6QHKYSJSNucTtj1BnzwCBzcAoMdMOc7rjVjIYP8XZ14SW7pUf6xJpcXN+RzpKaBccMGc9OcNC7OGK6eYyI+phAmIu2zFva+D6t+61ozNmgozP4WzPg6hA3xd3XiJccW8j+zOoedhyqJGRTM1TNTuH5WKo5o9RwT8QWFMBHxXO7HsOoR19qx0CEw8xsw61vqM9aPWGtZ83kZT6/ex7IdhQCcPyGJxXPSmJkeq6lKES9SCBORzjuwxTUylv1fCAqDyYvg9FshcYK/KxMvyi939Rz71/o8Dlc3MDYpisVz0rgkw0F4iKYqRbpLIUxEuq5oJ6z5I3zyL2ishbSz4PTbYMyFEKD/SfcXNfVN/GdLAU+7pyoHhwXxlemu7ZHSh0b4uzyRPkshTES6r7oMNj0D656CinyITnFtiTTtBgiP8Xd14iXWWtbtK+O5Nbm8ve0QjU7LWaOGcuPsNOaPTSAwQFOVIp2hECYi3tPU6Lqjcu2fXYv4gwfB5Ktco2MJY/1dnXhRUUUtL6zP4/m1+zlUUYsjOpxrT0/hqhkjGBoZ6u/yRPoEhTAR8Y2Dn8C6P8MnL0FTHaSfDdNuhDEL1eKiH2lscrI8u5Dn1uTy0Z5SggMNCycN48bZqUxLUc8xkfYohImIbx0thU1Pw/q/uaYqQyJh3MWuxfzpc7V2rB/ZU1TFP9bk8vLGfCrrXJuHXz8rlUunOojU5uEiJ1AIE5Ge4XRC7ofwyYuw4z9QVwGRSTDpSteUZdIk0KhJv1D9/9u789g4zvuM49/fLm9yeZ/iIZk6TMmWddqy46Sx5Ryu7dhJI+dwHKRFihRFCiRFizYpWhQNYPQC0gRogKZwg7qN2/iKbcV140MWnLixrcM6bFmSdZgSL5HiJR7iufv2j3dILmn51mqW5PMBBjPzznL17r7U7sN33nlnfJLH9rXz05dO8XrHAPlZUT67oZa7r13K6prCsKsnkjYUwkTk0psYgTee8oHs2NOQmICK1b53bO2dUFwfdg3lInDOsb+ln/tfPs0vDrQzNplgY0Mxd1+7lFvW1mhGfln0FMJEJFzne/29Kg8+CC0v+7L6LX7sWNOtUL4y3PrJRdF/fpxHXmnj/pdPcfLsMMV5mdy5qY67tmiaC1m8FMJEJH30vgmvPgyHt8OZg76sbKWfd6zpVqi7WmPI5jnnHC+e7OH+l07z1CE/zcX1K8q4c1M9n76iWpPAyqKiECYi6am/BY7+Lxx9Epp/DYlJf9/KVTdD0y3QeKOuspznugZHeXB3Cw/saaGld4RYdga3rath26Z6NjYU68pKWfAUwkQk/Y2eg2PP+FB27BkYO+dvl9R4Ayzf6gNZ+UoN7J+nlXUauQAAE6NJREFUEgnHy2/28vDeVp58tYORiTiNFfls21TH72yoo7ooJ+wqiqSEQpiIzC+T43D6N3DkSTj2FPQ1+/LCOlh+gw9kjTdAfnl4dZQPbGhskicPdvDQ3hZ2N/cRMfjYygq2barjk2uqNJhfFhSFMBGZ33rfhJM74cROePN532sGUH0VLL/Rh7KG6yBTvSnzTXP3MA/vbeWRV1rpODdKYU4Gd6yv5c7NdaytLdLpSpn3FMJEZOFIxKF9P5x4zgezll1++ouMHFiyEeo2B8vVULgk7NrKexRPOH5zopuH9rTyy0NnGJ9M0FQdY9umOj63oZYy3SZJ5imFMBFZuMaG/D0sTz4Prbug4wDEx/2xwtqZQFa7GZash8zccOsr7+rc+Qm2H2zn4T0tHGg9R0bEuGl1JXduqueGyyvIiEbCrqLIe6YQJiKLx+QYnHkVWvdA626/9J/yxyIZUHUl1G6CmnU+lFWshoyscOssb+vomUEe3tvCo/va6B4ap7wgm89v9KcrV1TGwq6eyLtSCBORxW3oLLQFoawl6C0bG/DHIplQtQZq1s8Es8orNL4szUzEE+w80sVDe1vZeaSLyYRjXV0Rt6+v5TNX1VBZqPaS9KQQJiKSLJGAvjehY78PZO3BerTfH49k+B6y6rVQvgLKVkDpciht1LxlaaB7aIzH9rXx2P42XmsbIGLwkeXl3L5+CTdfWU1hTmbYVRSZphAmIvJunIP+07ODWedrMNQ5+3GFtT6Mla2AsuU+nJWtgJJlOq0ZguNdQ2w/0M7j+9s41XOerIwIWy+v5I71S7ixqVLTXUjoFMJERD6o0QHoPQm9J6DnJPQcD7ZPwEjvzOMsAkV1QShbPtNzVrYcipcqoKWYc44Dred4fH8bvzjQQffQGLHsDD59ZTV3rF/CdY1lGtAvoVAIExFJhfO9PqD1HA/WJ2bC2ti5mcdZBIobfCgrWeanzogt8evCWiisgWwNMr9YJuMJXjzZw+P72/nla2cYGpukvCCLW9fWcPv6JWxsKNH8Y3LJKISJiFxKzgUBLegxm+5JO+FPeSb3oE3JLoRYzUwwi1VDfgXklUFeqb87QF6ZXzTNxns2OhFn55Euth9oZ8eRLsYnE9QW5/KZdUu4fd0SVtfEFMgkpRTCRETSycQIDHbAQDsMdMBAW7DfFuy3w9AZcIkL/3xm/uxwll3oe9KyY5BVEGwH66zYzH5OkQ92izTEDY5O8PShTrYfaOeF493EE44VlQXcHgSyZeX5YVdRUs05GB+C8z1+ySn2QwZSSCFMRGS+SST81ZpTXxbD3TPbyctwt59uY2wIxgZhYvjdnzsr5sNbfoVfCipmtvPLIa8csvL9XQgyc/0ytZ2RsyBuot4zNMaTr53hF/vb2dXseyavqivi01dUs7WpkqZq9ZDNC1P/T4a6YPgsDHe9zf+V3pntqcmcATZ/HW77fkqrqBAmIrJYJOL+L/2pUDYerMcGYaQPznf7L6nhs8EXV7B9vvvte95msSCQ5UBmng9r2YWQU+h72qa2s4uCdeEF1sHjohkpfzvei/b+EZ442M4TBzs42OrH8i0pymHr6kpuWl3FdY1li/Mqy8nxIOAP+N+f0QGYHPW/Jy7hf9emtl3c9zJNlePAohCJ+jGRkYxgOyhL3k5MQnzCT7QcH0vaHp9Zx8d9HYbPBr+7Z2d+bxOTF6i8QW7JzCn8qZ7jWftlvhesfGVK30aFMBEReWeJhA9pUz0JEyMwOQITo8E6WCZHk7ZHgrA34G+qPhp8YY8O+GPvJjNvTnALTqua+S903Jw1SftTQWAyWOJJ20n78Qn/RT/d61cJ+ZVB718lFFTNbGfm0DUwys6jXTx7uIsXjnUzMhEnNzPK9SvKuWl1JVubKqlK5cSwiXjw3o7593ByLHjPR/16csSHo8SEf23xCR9Q3rIfvPbEBMSn3pOJoPwC++NDM2FrKnDFx1L3Oj+IjJyktquY06ZJ23nlPnBF0iM4hxbCzOxm4IdAFLjXOfd3c47/LvCPQFtQ9M/OuXvf6TkVwkRE5oH4RBDK5oSztwS2c7OPjQ36oGUG2IXXU9uRaNDDkjHT0/KW/UwfNoa7g56/Lv9vXkhWDDKyIZoJkUwSkSgjcWNw3Ogfc4zGjQmiZGdlU5SfS2lBDgW5WZhFfG/P9GIz287N7tGZHJ2zPe7DzlTYumCvzgdkEf/6o5kz70UkY/b+1Pb0WMIgCOcUJu0nlWXkQiT5tUZntqd6vabaZ7q3LD4Til3cB34XnwnLkYyZ9z0arDOy52xnpU2oer/eKYSlrC/YzKLAj4BPAq3AbjPb7px7fc5DH3DO/VGq6iEiIiGIZkJ+mV/SzeRY0unYszPhbLjbHwt6jyKJCfLjE+QnJqlKTHB+ZJS+wfMMDI9wtq+fnr4E2VGjKCdKUU6U/KwIEdzs03TgQ8RUqMgq8KfBpvaTj2Xm+N6e6SU7GIeXPbs8mulDSTTLn9KNZs0EqqljkUwfliStpfKE/DXAcefcSQAz+xlwBzA3hImIiFw6Gdl+Yt2iuvf8IwbkBwv4gf07j57l2dc7+dWxs5zvi5OXFeVjK8v5xOoqtjZVUlaQnYraywKSyhBWC7Qk7bcCWy7wuM+b2W8BbwB/7JxrmfsAM/sG8A2AhoaGFFRVRETkvSsryGbbpjq2bapjdCLOiyd72HG4k2df7+KpQ52YwaaGEm4KAtmqqgJdbSlvkbIxYWa2DbjZOff7wf5XgS3Jpx7NrAwYcs6NmdkfAF90zm19p+fVmDAREUlXzjkOtQ/w7OFOnj3cyWttAwDUFueytckP7L9u+SK92nKRCmVMGH6wfX3Sfh0zA/ABcM71JO3eC/xDCusjIiKSUmbGlbVFXFlbxLc/sYqOcyPsPHKW54508fDeVv7zpVPkZEa4fnk5NwahbEnx4pw8V1IbwnYDK83sMnz4+hJwV/IDzKzGOdcR7N4OHE5hfURERC6pmqJc7trSwF1bGhidiPPym708d7iT5452seNIFwBN1TFubKrkI8vL2NhQQn52esyfJqmX6ikqbgF+gJ+i4ifOuXvM7HvAHufcdjP7W3z4mgR6gT90zh15p+fU6UgREZnvnHOcODvEjsNdPHekiz2n+ognHNGIsaamkM3LSrh6WSmbl5VQGUvhvGSScpqsVUREJI0Njk6w73Q/u5t72d3cy/6WfkYn/BQXy8ry2LyslGuCUHZZeb4G+c8jCmEiIiLzyPhkgkPt59jT3Meu5l72NPfSd34CgPKCLLY0lnFtYxnXNZayvEJXXqYzhTAREZF5zJ++HGZ3cy+73uzlpZM9dJwbBYJQdlkZ1zaWcm1jGSsqFcrSSVhXR4qIiMhFYGasqCxgRWUBX76mAeccLb0jvHSyZ3r5n1f9dW5l+VlsaSxly2VlbF5WQlN1IdGIQlk6UggTERGZZ8yMhrI8Gsry+MLV9TjnaO0b4cUgkL18spcnXz0DQEF2Bhsaitm01A/2X19frCsw04RaQUREZJ4zM+pL86gvzeMLm/0Una1959l7qo/dzb3sae7jhzuO4RxEI8bqmhibl5ayaWkJm5eVUFOkucrCoDFhIiIii8BAcAXmniCU7W/pZ2QiDkBNUQ7r6oq5qr6I9XXFrK0rIpaTGXKNFwaNCRMREVnkCnMy+fiqCj6+qgKAiXiCwx0D7Gnu40BrPwda+vnlIX8K0wyWVxSwrq6YdfVFrKsrpqkmRnaGbrd0MSmEiYiILEKZ0QhX1RVzVV3xdFnf8DgH285xsKWfA639PP9GF4+80gpAVjTCmiWFbGgoZmNDCRuXlrCkKEdXYn4IOh0pIiIiF+Sco/3cKAdafE/ZvpZ+DrbOTCRbVZjNhvoSNi71wezK2iLdnHwOnY4UERGR983MqC3OpbY4l1vW1gD+NOaRjkFeOd3HvtN9vHJ65jRmZtRYs6SIDfXFrKqKsbKqgJWVBRTnZYX5MtKWesJERETkQzk7ODYdyPad7uPVtnOcH49PHy8vyGZlMM/ZyqpgXRmjvCBrwZ/OVE+YiIiIpExFLJtPXVHNp66oBiCRcLSfG+FY1xDHO4c41jXIsa4hHtvXxuDY5PTPFedlsqoyxuXVMVZVx2iqjrGqKkZR7uK4MlMhTERERC6qSMSoK8mjriSPGy+vnC53ztE5MMaxrkGOdw3xRucQb3QOviWc1RTlcHl1jMurfEC7vDrG8oqCBTfeTCFMRERELgkzo7ooh+qiHD62smK6fOoCgKNnBjh6ZsivO4f4zfEexuOJ4GdhSVEujRX5XFaez7KyfC6ryKexPJ/a4lwyopGwXtYHphAmIiIioUq+AGBrU9V0+UQ8wameYY6cGeRY5xDNPcM0dw/z6L42Bkdnes4yo/6OAZeV5bOsPJ+G0jzqSnKpK8mjtiSXgjS9TVN61kpEREQWvcxohBWVMVZUxmaVO+foGR6nuXuYk90+mL0ZLP93ont6Co0pJXmZwenR3OlwVleSy6qqGPWleZfyJc2iECYiIiLziplRXpBNeUE2m5eVzjo2FdBa+0Zo6T1Pa98IrX1+/UbnIM8d6WJs0oe0r2xp4J7PrQ3jJQAKYSIiIrKAJAe09fXFbznunKN7aJzWvvPEcsKNQQphIiIismiYGRWxbCpi2WFXhfl3KYGIiIjIAqAQJiIiIhIChTARERGRECiEiYiIiIRAIUxEREQkBAphIiIiIiFQCBMREREJgUKYiIiISAgUwkRERERCoBAmIiIiEgKFMBEREZEQKISJiIiIhEAhTERERCQECmEiIiIiIVAIExEREQmBOefCrsP7YmZngVOX4J8qB7ovwb8j75/aJr2pfdKX2ia9qX3S14dpm6XOuYoLHZh3IexSMbM9zrnNYddD3kptk97UPulLbZPe1D7pK1Vto9ORIiIiIiFQCBMREREJgULY2/vXsCsgb0ttk97UPulLbZPe1D7pKyVtozFhIiIiIiFQT5iIiIhICBTCREREREKgEDaHmd1sZkfN7LiZfSfs+ix2ZvYTM+sys9eSykrN7BkzOxasS8Ks42JlZvVmttPMXjezQ2b2raBc7ZMGzCzHzHaZ2YGgff4mKL/MzF4OPuMeMLOssOu6WJlZ1Mz2mdkTwb7aJk2YWbOZvWpm+81sT1B20T/bFMKSmFkU+BHw28Aa4MtmtibcWi16/w7cPKfsO8AO59xKYEewL5feJPAnzrk1wLXAN4P/L2qf9DAGbHXOrQPWAzeb2bXA3wP/5JxbAfQBXw+xjovdt4DDSftqm/Ryo3NufdL8YBf9s00hbLZrgOPOuZPOuXHgZ8AdIddpUXPO/QronVN8B3BfsH0f8NlLWikBwDnX4Zx7JdgexH+Z1KL2SQvOGwp2M4PFAVuBh4NytU9IzKwOuBW4N9g31Dbp7qJ/timEzVYLtCTttwZlkl6qnHMdwfYZoCrMygiY2TJgA/Ayap+0EZzu2g90Ac8AJ4B+59xk8BB9xoXnB8CfAYlgvwy1TTpxwNNmttfMvhGUXfTPtowP+wQiYXLOOTPTPCshMrMC4BHg2865Af8Hvaf2CZdzLg6sN7Ni4FGgKeQqCWBmtwFdzrm9ZnZD2PWRC/qoc67NzCqBZ8zsSPLBi/XZpp6w2dqA+qT9uqBM0kunmdUABOuukOuzaJlZJj6A3e+c+3lQrPZJM865fmAncB1QbGZTf4DrMy4c1wO3m1kzftjLVuCHqG3ShnOuLVh34f+AuYYUfLYphM22G1gZXKGSBXwJ2B5yneSttgNfC7a/BjweYl0WrWAMy78Bh51z3086pPZJA2ZWEfSAYWa5wCfx4/Z2AtuCh6l9QuCc+65zrs45twz/PfOcc+4rqG3Sgpnlm1lsahv4FPAaKfhs04z5c5jZLfhz9VHgJ865e0Ku0qJmZv8N3ACUA53AXwOPAQ8CDcAp4AvOubmD9yXFzOyjwK+BV5kZ1/IX+HFhap+QmdlV+MHDUfwf3A86575nZo343pdSYB9wt3NuLLyaLm7B6cg/dc7dprZJD0E7PBrsZgD/5Zy7x8zKuMifbQphIiIiIiHQ6UgRERGRECiEiYiIiIRAIUxEREQkBAphIiIiIiFQCBMREREJgUKYiMh7ZGY3mNkTYddDRBYGhTARERGRECiEiciCY2Z3m9kuM9tvZj8ObmQ9ZGb/ZGaHzGyHmVUEj11vZi+Z2UEze9TMSoLyFWb2rJkdMLNXzGx58PQFZvawmR0xs/st+WaZIiLvg0KYiCwoZrYa+CJwvXNuPRAHvgLkA3ucc1cAz+PvvgDwH8CfO+euws/+P1V+P/Aj59w64CNAR1C+Afg2sAZoxN8HUETkfct494eIiMwrNwGbgN1BJ1Uu/ka7CeCB4DE/BX5uZkVAsXPu+aD8PuCh4L5xtc65RwGcc6MAwfPtcs61Bvv7gWXAC6l/WSKy0CiEichCY8B9zrnvzio0+6s5j/ug92xLvpdfHH2OisgHpNORIrLQ7AC2mVklgJmVmtlS/OfdtuAxdwEvOOfOAX1m9rGg/KvA8865QaDVzD4bPEe2meVd0lchIgue/oITkQXFOfe6mf0l8LSZRYAJ4JvAMHBNcKwLP24M4GvAvwQh6yTwe0H5V4Efm9n3gue48xK+DBFZBMy5D9ojLyIyf5jZkHOuIOx6iIhM0elIERERkRCoJ0xEREQkBOoJExEREQmBQpiIiIhICBTCREREREKgECYiIiISAoUwERERkRD8P04G/4v0KU1RAAAAAElFTkSuQmCC\n"},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","source":["model_outcomes = [original_test_file, original_test_file_excluding_last_two_chars,\n","                 mlp_pred, gru_pred]\n","df = pd.DataFrame (model_outcomes).transpose()\n","df.columns = ['Original Chars', 'Original Chars Excluding Last Two Letters',\n","               'MLP', 'GRU']\n","df"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":519},"id":"BC_LOyT2ZEwJ","executionInfo":{"status":"ok","timestamp":1673907147965,"user_tz":-60,"elapsed":506,"user":{"displayName":"Olusegun Ajibola","userId":"01030886262668976591"}},"outputId":"6e1652db-13ec-4976-cf94-326da55e9677"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["   Original Chars Original Chars Excluding Last Two Letters            MLP  \\\n","0    Biographical                                Biographic  Biographice\\r   \n","1     Wheelbarrow                                 Wheelbarr    Wheelbarr n   \n","2       Nightgown                                   Nightgo      Nightgore   \n","3     Counterpane                                 Counterpa    Counterpasi   \n","4       Christmas                                   Christm      Christm h   \n","5         Squires                                     Squir        Squirin   \n","6     Specksnyder                                 Specksnyd    Specksnyd s   \n","7      Forecastle                                  Forecast    Forecaste\\r   \n","8       Affidavit                                   Affidav     Affidave\\r   \n","9           Hyena                                       Hye          Hyeng   \n","10      Monstrous                                   Monstro      Monstroo    \n","11        Town-Ho                                     Town-        Town-be   \n","12       Massacre                                    Massac      Massack\\r   \n","13        Funeral                                     Funer        Funer,    \n","14  Battering-Ram                               Battering-R  Battering-R m   \n","\n","                GRU  \n","0      Biographical  \n","1       Wheelbarrow  \n","2         Nightgo,   \n","3       Counterpane  \n","4         Christmat  \n","5          Squir,\\r  \n","6       Specksnydly  \n","7        Forecast l  \n","8         Affidavis  \n","9             Hyers  \n","10        Monstrone  \n","11          Town-ya  \n","12         Massache  \n","13          Funeran  \n","14  Battering-R\\r\\n  "],"text/html":["\n","  <div id=\"df-da893e74-d03e-4473-8924-72a9d10c4b3a\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>Original Chars</th>\n","      <th>Original Chars Excluding Last Two Letters</th>\n","      <th>MLP</th>\n","      <th>GRU</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Biographical</td>\n","      <td>Biographic</td>\n","      <td>Biographice\\r</td>\n","      <td>Biographical</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Wheelbarrow</td>\n","      <td>Wheelbarr</td>\n","      <td>Wheelbarr n</td>\n","      <td>Wheelbarrow</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Nightgown</td>\n","      <td>Nightgo</td>\n","      <td>Nightgore</td>\n","      <td>Nightgo,</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Counterpane</td>\n","      <td>Counterpa</td>\n","      <td>Counterpasi</td>\n","      <td>Counterpane</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Christmas</td>\n","      <td>Christm</td>\n","      <td>Christm h</td>\n","      <td>Christmat</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>Squires</td>\n","      <td>Squir</td>\n","      <td>Squirin</td>\n","      <td>Squir,\\r</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>Specksnyder</td>\n","      <td>Specksnyd</td>\n","      <td>Specksnyd s</td>\n","      <td>Specksnydly</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>Forecastle</td>\n","      <td>Forecast</td>\n","      <td>Forecaste\\r</td>\n","      <td>Forecast l</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>Affidavit</td>\n","      <td>Affidav</td>\n","      <td>Affidave\\r</td>\n","      <td>Affidavis</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>Hyena</td>\n","      <td>Hye</td>\n","      <td>Hyeng</td>\n","      <td>Hyers</td>\n","    </tr>\n","    <tr>\n","      <th>10</th>\n","      <td>Monstrous</td>\n","      <td>Monstro</td>\n","      <td>Monstroo</td>\n","      <td>Monstrone</td>\n","    </tr>\n","    <tr>\n","      <th>11</th>\n","      <td>Town-Ho</td>\n","      <td>Town-</td>\n","      <td>Town-be</td>\n","      <td>Town-ya</td>\n","    </tr>\n","    <tr>\n","      <th>12</th>\n","      <td>Massacre</td>\n","      <td>Massac</td>\n","      <td>Massack\\r</td>\n","      <td>Massache</td>\n","    </tr>\n","    <tr>\n","      <th>13</th>\n","      <td>Funeral</td>\n","      <td>Funer</td>\n","      <td>Funer,</td>\n","      <td>Funeran</td>\n","    </tr>\n","    <tr>\n","      <th>14</th>\n","      <td>Battering-Ram</td>\n","      <td>Battering-R</td>\n","      <td>Battering-R m</td>\n","      <td>Battering-R\\r\\n</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-da893e74-d03e-4473-8924-72a9d10c4b3a')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-da893e74-d03e-4473-8924-72a9d10c4b3a button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-da893e74-d03e-4473-8924-72a9d10c4b3a');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":251}]}],"metadata":{"colab":{"provenance":[{"file_id":"https://github.com/tensorflow/text/blob/master/docs/tutorials/text_generation.ipynb","timestamp":1673849595397}]},"kernelspec":{"display_name":"Python 3","name":"python3"},"gpuClass":"standard","accelerator":"GPU"},"nbformat":4,"nbformat_minor":0}