Deciphering Code with Character-Level RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deciphering Code with Character-Level RNN\n",
    "\n",
    "In this notebook, we'll look at how to build a recurrent neural network and train it to decipher strings encrypted with a certain cipher.\n",
    "\n",
    "This exercise will make you familiar with the techniques of preprocessing and model-building that will come in handy when you start building more advanced models for machine translation, text summarization, and beyond.\n",
    "\n",
    "## Dataset\n",
    "The dataset we have consists of 10,000 encrypted phrases and the plaintext version of each encrypted phrase.\n",
    "\n",
    "Let's start by loading up the dataset to get more familiar with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import helper\n",
    "\n",
    "codes = helper.load_data('cipher.txt')\n",
    "plaintext = helper.load_data('plaintext.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now `codes` and `plaintext` are both arrays with each element being a phrase. The first three encoded phrases are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['YMJ QNRJ NX MJW QJFXY QNPJI KWZNY , GZY YMJ GFSFSF NX RD QJFXY QNPJI .',\n",
       " 'MJ XFB F TQI DJQQTB YWZHP .',\n",
       " 'NSINF NX WFNSD IZWNSL OZSJ , FSI NY NX XTRJYNRJX BFWR NS STAJRGJW .',\n",
       " 'YMFY HFY BFX RD RTXY QTAJI FSNRFQ .',\n",
       " 'MJ INXQNPJX LWFUJKWZNY , QNRJX , FSI QJRTSX .']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "codes[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And their plaintext versions are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['THE LIME IS HER LEAST LIKED FRUIT , BUT THE BANANA IS MY LEAST LIKED .',\n",
       " 'HE SAW A OLD YELLOW TRUCK .',\n",
       " 'INDIA IS RAINY DURING JUNE , AND IT IS SOMETIMES WARM IN NOVEMBER .',\n",
       " 'THAT CAT WAS MY MOST LOVED ANIMAL .',\n",
       " 'HE DISLIKES GRAPEFRUIT , LIMES , AND LEMONS .']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plaintext[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Overview: Character-Level RNN\n",
    "The model we will use here is a character-level RNN since the cipher seems to work on the characer level. In a machine translation scenario, a word-level RNN is the more common choice.\n",
    "\n",
    "A character-level RNN will take as input an integer referring to a specific character and output another integer. To be able to get our model to work, we'll need to preprocess our dataset in the following steps:\n",
    " 1. Isolating each character as an array element (instead of an entire phrase, or word being the element of the array)\n",
    " 1. Tokenizing the characters so we can turn them from letters to integers and vice-versa\n",
    " 1. Padding the strings so that all the inputs and outputs can fit in matrix form\n",
    " \n",
    "To visualize this processing, let's assume either our source sequences (`codes` in this case) or target sequences (`plaintext` in this case) look like this (a list of strings):\n",
    "\n",
    "<img src=\"list_1.png\" />\n",
    "\n",
    "Since this model will be working on the character level, we'll need to separate each string into a list of characters (implicitly done by the tokenizer in this notebook):\n",
    "\n",
    "<img src=\"list_2.png\" />\n",
    "\n",
    "Then, the process of tokenization will turn each character into an integer.  Note that when you're working on the a word-level RNN (as in most machine translation examples), the tokenizer will assign an integer to each word rather than each letter, and each cell would represent a word rather than a character.\n",
    "\n",
    "<img src=\"list_3.png\" />\n",
    "\n",
    "Most machine learning platforms expect the input to be a matrix rather than a list of lists. To turn the input into a matrix, we need to find the longest member of the list, and pad all shorter sequences with 0. Assuming 'and two' is the longest sequence in this example, the matrix ends up looking like this:\n",
    "\n",
    "<img src=\"padded_list.png\" />\n",
    " \n",
    "## Preprocessing (IMPLEMENT)\n",
    "For a neural network to predict on text data, it first has to be turned into data it can understand. Text data like \"dog\" is a sequence of ASCII character encodings.  Since a neural network is a series of multiplication and addition operations, the input data needs to be number(s).\n",
    "\n",
    "We can turn each character into a number or each word into a number.  These are called character and word ids, respectively.  Character ids are used for character level models that generate text predictions for each character.  A word level model uses word ids that generate text predictions for each word.  Word level models tend to learn better.\n",
    "\n",
    "Turn each sentence into a sequence of words ids using Keras's [`Tokenizer`](https://keras.io/preprocessing/text/#tokenizer) function. Since we're working on the character level, make sure to set the `char_level` flag to the appropriate value. Then, fit the tokenizer on x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{' ': 1, 'e': 2, 'o': 3, 'i': 4, 's': 5, 'h': 6, 'r': 7, 'y': 8, 'u': 9, 'c': 10, 'n': 11, 't': 12, 'a': 13, 'p': 14, '.': 15, 'T': 16, 'q': 17, 'k': 18, 'w': 19, 'f': 20, 'x': 21, 'm': 22, 'v': 23, 'l': 24, 'z': 25, 'd': 26, 'g': 27, 'b': 28, 'j': 29, 'B': 30, 'J': 31, ',': 32}\n",
      "\n",
      "Sequence 1 in x\n",
      "  Input:  The quick brown fox jumps over the lazy dog .\n",
      "  Output: [16, 6, 2, 1, 17, 9, 4, 10, 18, 1, 28, 7, 3, 19, 11, 1, 20, 3, 21, 1, 29, 9, 22, 14, 5, 1, 3, 23, 2, 7, 1, 12, 6, 2, 1, 24, 13, 25, 8, 1, 26, 3, 27, 1, 15]\n",
      "Sequence 2 in x\n",
      "  Input:  By Jove , my quick study of lexicography won a prize .\n",
      "  Output: [30, 8, 1, 31, 3, 23, 2, 1, 32, 1, 22, 8, 1, 17, 9, 4, 10, 18, 1, 5, 12, 9, 26, 8, 1, 3, 20, 1, 24, 2, 21, 4, 10, 3, 27, 7, 13, 14, 6, 8, 1, 19, 3, 11, 1, 13, 1, 14, 7, 4, 25, 2, 1, 15]\n",
      "Sequence 3 in x\n",
      "  Input:  This is a short sentence .\n",
      "  Output: [16, 6, 4, 5, 1, 4, 5, 1, 13, 1, 5, 6, 3, 7, 12, 1, 5, 2, 11, 12, 2, 11, 10, 2, 1, 15]\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "\n",
    "def tokenize(x):\n",
    "    \"\"\"\n",
    "    Tokenize x\n",
    "    :param x: List of sentences/strings to be tokenized\n",
    "    :return: Tuple of (tokenized x data, tokenizer used to tokenize x)\n",
    "    \"\"\"\n",
    "    # TODO: Implement\n",
    "    x_tk = Tokenizer(char_level=True)\n",
    "    x_tk.fit_on_texts(x)\n",
    "\n",
    "    return x_tk.texts_to_sequences(x), x_tk\n",
    "\n",
    "# Tokenize Example output\n",
    "text_sentences = [\n",
    "    'The quick brown fox jumps over the lazy dog .',\n",
    "    'By Jove , my quick study of lexicography won a prize .',\n",
    "    'This is a short sentence .']\n",
    "text_tokenized, text_tokenizer = tokenize(text_sentences)\n",
    "print(text_tokenizer.word_index)\n",
    "print()\n",
    "for sample_i, (sent, token_sent) in enumerate(zip(text_sentences, text_tokenized)):\n",
    "    print('Sequence {} in x'.format(sample_i + 1))\n",
    "    print('  Input:  {}'.format(sent))\n",
    "    print('  Output: {}'.format(token_sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Padding (IMPLEMENTATION)\n",
    "When batching the sequence of word ids together, each sequence needs to be the same length.  Since sentences are dynamic in length, we can add padding to the end of the sequences to make them the same length.\n",
    "\n",
    "Make sure all the cipher sequences have the same length and all the plaintext sequences have the same length by adding padding to the **end** of each sequence using Keras's [`pad_sequences`](https://keras.io/preprocessing/sequence/#pad_sequences) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequence 1 in x\n",
      "  Input:  [16  6  2  1 17  9  4 10 18  1 28  7  3 19 11  1 20  3 21  1 29  9 22 14  5\n",
      "  1  3 23  2  7  1 12  6  2  1 24 13 25  8  1 26  3 27  1 15]\n",
      "  Output: [16  6  2  1 17  9  4 10 18  1 28  7  3 19 11  1 20  3 21  1 29  9 22 14  5\n",
      "  1  3 23  2  7  1 12  6  2  1 24 13 25  8  1 26  3 27  1 15  0  0  0  0  0\n",
      "  0  0  0  0]\n",
      "Sequence 2 in x\n",
      "  Input:  [30  8  1 31  3 23  2  1 32  1 22  8  1 17  9  4 10 18  1  5 12  9 26  8  1\n",
      "  3 20  1 24  2 21  4 10  3 27  7 13 14  6  8  1 19  3 11  1 13  1 14  7  4\n",
      " 25  2  1 15]\n",
      "  Output: [30  8  1 31  3 23  2  1 32  1 22  8  1 17  9  4 10 18  1  5 12  9 26  8  1\n",
      "  3 20  1 24  2 21  4 10  3 27  7 13 14  6  8  1 19  3 11  1 13  1 14  7  4\n",
      " 25  2  1 15]\n",
      "Sequence 3 in x\n",
      "  Input:  [16  6  4  5  1  4  5  1 13  1  5  6  3  7 12  1  5  2 11 12  2 11 10  2  1\n",
      " 15]\n",
      "  Output: [16  6  4  5  1  4  5  1 13  1  5  6  3  7 12  1  5  2 11 12  2 11 10  2  1\n",
      " 15  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "\n",
    "def pad(x, length=None):\n",
    "    \"\"\"\n",
    "    Pad x\n",
    "    :param x: List of sequences.\n",
    "    :param length: Length to pad the sequence to.  If None, use length of longest sequence in x.\n",
    "    :return: Padded numpy array of sequences\n",
    "    \"\"\"\n",
    "    # TODO: Implement\n",
    "    # Find the length of the longest string in the dataset. \n",
    "    # Then, pass it to pad_sentences as the maxlen parameter\n",
    "    if length is None:\n",
    "        length = max([len(i) for i in x])\n",
    "    return pad_sequences(x, maxlen=length, padding='post')\n",
    "    \n",
    "    return \n",
    "\n",
    "# Pad Tokenized output\n",
    "test_pad = pad(text_tokenized)\n",
    "for sample_i, (token_sent, pad_sent) in enumerate(zip(text_tokenized, test_pad)):\n",
    "    print('Sequence {} in x'.format(sample_i + 1))\n",
    "    print('  Input:  {}'.format(np.array(token_sent)))\n",
    "    print('  Output: {}'.format(pad_sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess Pipeline\n",
    "Your focus for this project is to build neural network architecture, so we won't ask you to create a preprocess pipeline.  Instead, we've provided you with the implementation of the `preprocess` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Preprocessed\n"
     ]
    }
   ],
   "source": [
    "def preprocess(x, y):\n",
    "    \"\"\"\n",
    "    Preprocess x and y\n",
    "    :param x: Feature List of sentences\n",
    "    :param y: Label List of sentences\n",
    "    :return: Tuple of (Preprocessed x, Preprocessed y, x tokenizer, y tokenizer)\n",
    "    \"\"\"\n",
    "    preprocess_x, x_tk = tokenize(x)\n",
    "    preprocess_y, y_tk = tokenize(y)\n",
    "\n",
    "    preprocess_x = pad(preprocess_x)\n",
    "    preprocess_y = pad(preprocess_y)\n",
    "\n",
    "    # Keras's sparse_categorical_crossentropy function requires the labels to be in 3 dimensions\n",
    "    preprocess_y = preprocess_y.reshape(*preprocess_y.shape, 1)\n",
    "\n",
    "    return preprocess_x, preprocess_y, x_tk, y_tk\n",
    "\n",
    "preproc_code_sentences, preproc_plaintext_sentences, code_tokenizer, plaintext_tokenizer =\\\n",
    "    preprocess(codes, plaintext)\n",
    "\n",
    "print('Data Preprocessed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5, 14,  3,  1, 10,  2, 13,  3,  1,  2,  4,  1, 14,  3,  6,  1, 10,\n",
       "         3,  8,  4,  5,  1, 10,  2, 25,  3, 11,  1, 20,  6,  9,  2,  5,  1,\n",
       "        18,  1, 17,  9,  5,  1,  5, 14,  3,  1, 17,  8,  7,  8,  7,  8,  1,\n",
       "         2,  4,  1, 13, 15,  1, 10,  3,  8,  4,  5,  1, 10,  2, 25,  3, 11,\n",
       "         1, 19,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
       "         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [14,  3,  1,  4,  8, 24,  1,  8,  1, 12, 10, 11,  1, 15,  3, 10, 10,\n",
       "        12, 24,  1,  5,  6,  9, 22, 25,  1, 19,  0,  0,  0,  0,  0,  0,  0,\n",
       "         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
       "         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
       "         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
       "         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]], dtype=int32)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preproc_code_sentences[0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import GRU, Input, Dense, TimeDistributed\n",
    "from keras.models import Model\n",
    "from keras.layers import Activation\n",
    "from keras.optimizers import Adam\n",
    "from keras.losses import sparse_categorical_crossentropy\n",
    "\n",
    "\n",
    "def simple_model(input_shape, output_sequence_length, code_vocab_size, plaintext_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train a basic RNN on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param code_vocab_size: Number of unique code characters in the dataset\n",
    "    :param plaintext_vocab_size: Number of unique plaintext characters in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    # TODO: Build the model\n",
    "\n",
    "    learning_rate = 1e-3\n",
    "\n",
    "    input_seq = Input(input_shape[1:])\n",
    "    rnn = GRU(64, return_sequences=True)(input_seq)\n",
    "    logits = TimeDistributed(Dense(plaintext_vocab_size))(rnn)\n",
    "\n",
    "    model = Model(input_seq, Activation('softmax')(logits))\n",
    "    model.compile(loss=sparse_categorical_crossentropy,\n",
    "                  optimizer=Adam(learning_rate),\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    \n",
    "    \n",
    "    return model\n",
    "\n",
    "\n",
    "# Reshaping the input to work with a basic RNN\n",
    "tmp_x = pad(preproc_code_sentences, preproc_plaintext_sentences.shape[1])\n",
    "tmp_x = tmp_x.reshape((-1, preproc_plaintext_sentences.shape[-2], 1))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the neural network\n",
    "simple_rnn_model = simple_model(\n",
    "    tmp_x.shape,\n",
    "    preproc_plaintext_sentences.shape[1],\n",
    "    len(code_tokenizer.word_index)+1,\n",
    "    len(plaintext_tokenizer.word_index)+1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 8000 samples, validate on 2001 samples\n",
      "Epoch 1/4\n",
      "8000/8000 [==============================] - 37s 5ms/step - loss: 1.5462 - acc: 0.5776 - val_loss: 0.9500 - val_acc: 0.7391\n",
      "Epoch 2/4\n",
      "8000/8000 [==============================] - 37s 5ms/step - loss: 0.7150 - acc: 0.8161 - val_loss: 0.5377 - val_acc: 0.8698\n",
      "Epoch 3/4\n",
      "8000/8000 [==============================] - 37s 5ms/step - loss: 0.4264 - acc: 0.9042 - val_loss: 0.3417 - val_acc: 0.9231\n",
      "Epoch 4/4\n",
      "8000/8000 [==============================] - 37s 5ms/step - loss: 0.2825 - acc: 0.9389 - val_loss: 0.2363 - val_acc: 0.9480\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f2e836b5c88>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_rnn_model.fit(tmp_x, preproc_plaintext_sentences, batch_size=32, epochs=4, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "`logits_to_text` function loaded.\n",
      "T H E   L I M E   I S   M E R   L E A S T   L I K E D   F R U I T   ,   B U T   T H E   . A N A N A   I S   M H   L E A S T   L I K E D   . <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "def logits_to_text(logits, tokenizer):\n",
    "    \"\"\"\n",
    "    Turn logits from a neural network into text using the tokenizer\n",
    "    :param logits: Logits from a neural network\n",
    "    :param tokenizer: Keras Tokenizer fit on the labels\n",
    "    :return: String that represents the text of the logits\n",
    "    \"\"\"\n",
    "    index_to_words = {id: word for word, id in tokenizer.word_index.items()}\n",
    "    index_to_words[0] = '<PAD>'\n",
    "\n",
    "    return ' '.join([index_to_words[prediction] for prediction in np.argmax(logits, 1)])\n",
    "\n",
    "print('`logits_to_text` function loaded.')\n",
    "\n",
    "print(logits_to_text(simple_rnn_model.predict(tmp_x[:1])[0], plaintext_tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'THE LIME IS HER LEAST LIKED FRUIT , BUT THE BANANA IS MY LEAST LIKED .'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plaintext[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And there it is. The RNN was able to learn this basic character-level cipher (which was a simple [Caesar cipher](https://en.wikipedia.org/wiki/Caesar_cipher). If you want a bigger cryptography challenge, check out [Learning the Enigma with Recurrent Neural Networks](https://greydanus.github.io/2017/01/07/enigma-rnn/). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}