l09c02_nlp_padding.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "l09c02_nlp_padding.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "punL79CN7Ox6"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "_ckMIh7O7s6D"
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RX9Yx50TUies"
      },
      "source": [
        "# Preparing text to use with TensorFlow models\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5Uhzt6vVIB2"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l09c02_nlp_padding.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l09c02_nlp_padding.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5_MCdtjT-bly"
      },
      "source": [
        "The high level steps to prepare text to be used in a machine learning model are:\n",
        "\n",
        "1.   Tokenize the words to get numerical values for them\n",
        "2.   Create numerical sequences of the sentences\n",
        "3.   Adjust the sequences to all be the same length.\n",
        "\n",
        "In this colab, you learn how to use padding to make the sequences all be the same length."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qJsd8KslUn7j"
      },
      "source": [
        "## Import the classes you need"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_ZxQf11OUtQI"
      },
      "source": [
        "# Import Tokenizer and pad_sequences\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1MeEgRq4WX0v"
      },
      "source": [
        "## Write some sentences\n",
        "\n",
        "Feel free to write your own sentences here.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PwM7IP2lTr7T",
        "outputId": "6c1112ba-b030-4800-cec4-8282ba8ae6a4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "sentences = [\n",
        "    'My favorite food is ice cream',\n",
        "    'do you like ice cream too?',\n",
        "    'My dog likes ice cream!',\n",
        "    \"your favorite flavor of icecream is chocolate\",\n",
        "    \"chocolate isn't good for dogs\",\n",
        "    \"your dog, your cat, and your parrot prefer broccoli\"\n",
        "]\n",
        "print(sentences)\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['My favorite food is ice cream', 'do you like ice cream too?', 'My dog likes ice cream!', 'your favorite flavor of icecream is chocolate', \"chocolate isn't good for dogs\", 'your dog, your cat, and your parrot prefer broccoli']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jaRa9opNWmA7"
      },
      "source": [
        "## Create the Tokenizer and define an out of vocabulary token\n",
        "When creating the Tokenizer, you can specify the max number of words in the dictionary. You can also specify a token to represent words that are out of the vocabulary (OOV), in other words, that are not in the dictionary. This OOV token will be used when you create sequences for sentences that contain words that are not in the word index."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P7wuOJaBWiHZ"
      },
      "source": [
        "tokenizer = Tokenizer(num_words = 100, oov_token=\"<OOV>\")"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r7nILrhKXPge"
      },
      "source": [
        "## Tokenize the words"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YXooiuwrXROU",
        "outputId": "e2361bef-394e-4de9-ff58-f5d870fce8d4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "tokenizer.fit_on_texts(sentences)\n",
        "word_index = tokenizer.word_index\n",
        "print(word_index)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'<OOV>': 1, 'your': 2, 'ice': 3, 'cream': 4, 'my': 5, 'favorite': 6, 'is': 7, 'dog': 8, 'chocolate': 9, 'food': 10, 'do': 11, 'you': 12, 'like': 13, 'too': 14, 'likes': 15, 'flavor': 16, 'of': 17, 'icecream': 18, \"isn't\": 19, 'good': 20, 'for': 21, 'dogs': 22, 'cat': 23, 'and': 24, 'parrot': 25, 'prefer': 26, 'broccoli': 27}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0U-oe201Xm7T"
      },
      "source": [
        "## Turn sentences into sequences \n",
        "\n",
        "Each word now has a unique number in the word index.  However, words in a sentence are in a specific order. You can't just randomly mix up words and have the outcome be a sentence.\n",
        "\n",
        "For example, although \"chocolate isn't good for dogs\" is a perfectly fine sentence, \"dogs isn't for chocolate good\" does not make sense as a sentence.\n",
        "\n",
        "So the next step to representing text in a way that can be meaningfully used by machine learning programs is to create numerical sequences that represent the sentences in the text.\n",
        "\n",
        "Each sentence will be converted into a sequence where each word is replaced by its number in the word index. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "70l5x1XRXoV4",
        "outputId": "4d92bb48-aefa-49f5-e02a-8dea387093fb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "sequences = tokenizer.texts_to_sequences(sentences)\n",
        "print (sequences)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[5, 6, 10, 7, 3, 4], [11, 12, 13, 3, 4, 14], [5, 8, 15, 3, 4], [2, 6, 16, 17, 18, 7, 9], [9, 19, 20, 21, 22], [2, 8, 2, 23, 24, 2, 25, 26, 27]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tcFghvQ34cZK"
      },
      "source": [
        "## Make the sequences all the same length\n",
        "\n",
        "Later, when you feed the sequences into a neural network to train a model, the sequences all need to be uniform in size. Currently the sequences have varied lengths, so the next step is to make them all be the same size, either by padding them with zeros and/or truncating them.\n",
        "\n",
        "Use f.keras.preprocessing.sequence.pad_sequences to add zeros to the sequences to make them all be the same length. By default, the padding goes at the start of the sequences, but you can specify to pad at the end.\n",
        "\n",
        "You can optionally specify the maximum length to pad the sequences to. Sequences that are longer than the specified max length will be truncated. By default, sequences are truncated from the beginning of the sequence, but you can specify to truncate from the end.\n",
        "\n",
        "If you don't provide the max length, then the sequences are padded to match the length of the longest sentence.\n",
        "\n",
        "For all the options when padding and truncating sequences, see https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/pad_sequences\n",
        "\n",
        " "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r0m_nqHg4gqu",
        "outputId": "a916247e-a57d-4d3c-a8b1-f52b16ddf0e1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "padded = pad_sequences(sequences)\n",
        "print(\"\\nWord Index = \" , word_index)\n",
        "print(\"\\nSequences = \" , sequences)\n",
        "print(\"\\nPadded Sequences:\")\n",
        "print(padded)\n"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Word Index =  {'<OOV>': 1, 'your': 2, 'ice': 3, 'cream': 4, 'my': 5, 'favorite': 6, 'is': 7, 'dog': 8, 'chocolate': 9, 'food': 10, 'do': 11, 'you': 12, 'like': 13, 'too': 14, 'likes': 15, 'flavor': 16, 'of': 17, 'icecream': 18, \"isn't\": 19, 'good': 20, 'for': 21, 'dogs': 22, 'cat': 23, 'and': 24, 'parrot': 25, 'prefer': 26, 'broccoli': 27}\n",
            "\n",
            "Sequences =  [[5, 6, 10, 7, 3, 4], [11, 12, 13, 3, 4, 14], [5, 8, 15, 3, 4], [2, 6, 16, 17, 18, 7, 9], [9, 19, 20, 21, 22], [2, 8, 2, 23, 24, 2, 25, 26, 27]]\n",
            "\n",
            "Padded Sequences:\n",
            "[[ 0  0  0  5  6 10  7  3  4]\n",
            " [ 0  0  0 11 12 13  3  4 14]\n",
            " [ 0  0  0  0  5  8 15  3  4]\n",
            " [ 0  0  2  6 16 17 18  7  9]\n",
            " [ 0  0  0  0  9 19 20 21 22]\n",
            " [ 2  8  2 23 24  2 25 26 27]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VzbGtYWQ6Ofo",
        "outputId": "63d64e43-942e-48ea-90ef-ef7c2d05450f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Specify a max length for the padded sequences\n",
        "padded = pad_sequences(sequences, maxlen=15)\n",
        "print(padded)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[ 0  0  0  0  0  0  0  0  0  5  6 10  7  3  4]\n",
            " [ 0  0  0  0  0  0  0  0  0 11 12 13  3  4 14]\n",
            " [ 0  0  0  0  0  0  0  0  0  0  5  8 15  3  4]\n",
            " [ 0  0  0  0  0  0  0  0  2  6 16 17 18  7  9]\n",
            " [ 0  0  0  0  0  0  0  0  0  0  9 19 20 21 22]\n",
            " [ 0  0  0  0  0  0  2  8  2 23 24  2 25 26 27]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HzkbHi0B64w8",
        "outputId": "ac277dbc-da2c-4032-e71c-be3128c92228",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Put the padding at the end of the sequences\n",
        "padded = pad_sequences(sequences, maxlen=15, padding=\"post\")\n",
        "print(padded)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[ 5  6 10  7  3  4  0  0  0  0  0  0  0  0  0]\n",
            " [11 12 13  3  4 14  0  0  0  0  0  0  0  0  0]\n",
            " [ 5  8 15  3  4  0  0  0  0  0  0  0  0  0  0]\n",
            " [ 2  6 16 17 18  7  9  0  0  0  0  0  0  0  0]\n",
            " [ 9 19 20 21 22  0  0  0  0  0  0  0  0  0  0]\n",
            " [ 2  8  2 23 24  2 25 26 27  0  0  0  0  0  0]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OLHheI477okX",
        "outputId": "e54ce09e-9d50-465c-bd87-2a724cd20c5d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Limit the length of the sequences, you will see some sequences get truncated\n",
        "padded = pad_sequences(sequences, maxlen=3)\n",
        "print(padded)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[ 7  3  4]\n",
            " [ 3  4 14]\n",
            " [15  3  4]\n",
            " [18  7  9]\n",
            " [20 21 22]\n",
            " [25 26 27]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OnRKDsR197-J"
      },
      "source": [
        "## What happens if some of the sentences contain words that are not in the word index?\n",
        "\n",
        "Here's where the \"out of vocabulary\" token is used. Try generating sequences for some sentences that have words that are not in the word index."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iqodOpn64c2U",
        "outputId": "8b611fa9-5cb1-42e3-b316-d616ba8eab62",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Try turning sentences that contain words that \n",
        "# aren't in the word index into sequences.\n",
        "# Add your own sentences to the test_data\n",
        "test_data = [\n",
        "    \"my best friend's favorite ice cream flavor is strawberry\",\n",
        "    \"my dog's best friend is a manatee\"\n",
        "]\n",
        "print (test_data)\n",
        "\n",
        "# Remind ourselves which number corresponds to the\n",
        "# out of vocabulary token in the word index\n",
        "print(\"<OOV> has the number\", word_index['<OOV>'], \"in the word index.\")\n",
        "\n",
        "# Convert the test sentences to sequences\n",
        "test_seq = tokenizer.texts_to_sequences(test_data)\n",
        "print(\"\\nTest Sequence = \", test_seq)\n",
        "\n",
        "# Pad the new sequences\n",
        "padded = pad_sequences(test_seq, maxlen=10)\n",
        "print(\"\\nPadded Test Sequence: \")\n",
        "\n",
        "# Notice that \"1\" appears in the sequence wherever there's a word \n",
        "# that's not in the word index\n",
        "print(padded)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[\"my best friend's favorite ice cream flavor is strawberry\", \"my dog's best friend is a manatee\"]\n",
            "<OOV> has the number 1 in the word index.\n",
            "\n",
            "Test Sequence =  [[5, 1, 1, 6, 3, 4, 16, 7, 1], [5, 1, 1, 1, 7, 1, 1]]\n",
            "\n",
            "Padded Test Sequence: \n",
            "[[ 0  5  1  1  6  3  4 16  7  1]\n",
            " [ 0  0  0  5  1  1  1  7  1  1]]\n"
          ]
        }
      ]
    }
  ]
}