Ex10.One-hot-encoding-of-words-or-characters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.th-koeln.de/img/logo.svg\" style=\"float: right;\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10th exercise: <font color=\"#C70039\">One-hot encodings of words or characters</font>\n",
    "* Course: DIS21a.1\n",
    "* Lecturer: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Author of notebook modifications and adaptations: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Date:   05.08.2021\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/674/1*YEJf9BQQh0ma1ECs6x_7yQ.png\" style=\"float: center;\" width=\"500\">\n",
    "\n",
    "---------------------------------\n",
    "**GENERAL NOTE 1**: \n",
    "Please make sure you are reading the entire notebook, since it contains a lot of information about your tasks (e.g. regarding the set of certain paramaters or specific computational tricks, etc.), and the written mark downs as well as comments contain a lot of information on how things work together as a whole. \n",
    "\n",
    "**GENERAL NOTE 2**: \n",
    "* Please, when commenting source code, just use English language only. \n",
    "* When describing an observation (for instance, after you have run through your test plan) you may use German language.\n",
    "This applies to all exercises in DIS 21a.1.  \n",
    "\n",
    "---------------------\n",
    "\n",
    "### <font color=\"ce33ff\">DESCRIPTION</font>:\n",
    "\n",
    "This notebook allows for learning how to deal with text data as this knowledge is needed when it comes to modeling sequences (e.g. spoken language).\n",
    "\n",
    "One-hot encoding is the most common and most basic way to turn a token (word or character) into a vector. It was used in the previous IMDB and Reuters classification examples and in both cases it was done with entire words. \n",
    "One-hot encoding is fairly simple and consists in associating a unique integer index to every word, then turning this integer index i into a binary vector of size N, which is the size of the vocabulary, that would be all-zeros except for the i-th entry, which would be 1. \n",
    "\n",
    "Of course, one-hot encoding can be done on character level as well. This exercise shows how one-hot encoding is implemented and this is what you will learn. So, here are examples of one-hot encoding: one for words, the other for characters. The third example shows how to implement this not from scrach, but using the built-in utilities in Keras.\n",
    "\n",
    "-------------------------------------------------------------------------------------------------------------\n",
    "\n",
    "### <font color=\"FFC300\">TASKS</font>:\n",
    "The tasks that you need to work on in this notebook are always indicated below as bullet points. \n",
    "If a task is more challenging and consists of several steps, this is indicated as well. \n",
    "Make sure you have worked down the task list and commented your doings. \n",
    "This should be done by using markdown.<br> \n",
    "<font color=red>Make sure you don't forget to specify your name and your matriculation number in the notebook before submitting it.</font>\n",
    "\n",
    "**YOUR TASKS in this exercise are as follows**:\n",
    "1. import the notebook to Google Colab.\n",
    "2. make sure you specified you name and your matriculation number in the header below my name and date. \n",
    "    * set the date too and remove mine.\n",
    "3. read the entire notebook carefully. \n",
    "    * add comments whereever you feel it necessary for better understanding.\n",
    "    * run the notebook and try to follow and understand all steps and examples.\n",
    "4. the exercise in this notebook contains one single implementational task. \n",
    "    * implement a one-hot encoding on character level using Keras' built-in functions. \n",
    "    * let yourself be guided from the other three examples in this notebook. \n",
    "   \n",
    "-----------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# START OF THE NOTEBOOK CODE\n",
    "----------------------------------------------------------------------------------------------------------------------\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"#C70039\">EXAMPLE I</font>\n",
    "## Word level one-hot encoding example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### necessary imports\n",
    "others are going to be included as soon as they are needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.2.4-tf'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow\n",
    "tensorflow.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the initial data. One entry per \"sample\". In this example, a \"sample\" is just a sentence, but also it could be an entire document. The first step is building an index by simply tokenizing the samples via the `split` method. In real life, punctuation and special characters would be stripped from the samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 10, 11)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# sample data\n",
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "\n",
    "# build an index of all tokens in the data\n",
    "token_index = {}\n",
    "for sample in samples:\n",
    "    # tokenize the samples via the `split` method\n",
    "    for word in sample.split():\n",
    "        if word not in token_index:\n",
    "            # assign a unique index to each unique word\n",
    "            token_index[word] = len(token_index) + 1\n",
    "            ''' IMPORTANT NOTE: ''' # index 0 is not used for anything.\n",
    "\n",
    "# Now vectorize the samples\n",
    "# Consider the first `max_length` words in each sample\n",
    "max_length = 10\n",
    "\n",
    "# store the results in a numpy array\n",
    "results = np.zeros((len(samples), max_length, max(token_index.values()) + 1))\n",
    "print(results.shape)\n",
    "\n",
    "for i, sample in enumerate(samples):\n",
    "    for j, word in list(enumerate(sample.split()))[:max_length]:\n",
    "        index = token_index.get(word)\n",
    "        results[i, j, index] = 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------------\n",
    "## <font color=\"#C70039\">EXAMPLE II</font>\n",
    "## Character level one-hot encoding example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "# take all printable ASCII characters\n",
    "characters = string.printable  \n",
    "token_index = dict(zip(characters, range(1, len(characters) + 1)))\n",
    "\n",
    "max_length = 50\n",
    "results = np.zeros((len(samples), max_length, max(token_index.values()) + 1))\n",
    "\n",
    "for i, sample in enumerate(samples):\n",
    "    for j, character in enumerate(sample[:max_length]):\n",
    "        index = token_index.get(character)\n",
    "        results[i, j, index] = 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------------\n",
    "## <font color=\"#C70039\">EXAMPLE III</font>\n",
    "## Word and character level one-hot encoding example with Keras' built-in functions \n",
    "\n",
    "In Keras there are built-in functions for one-hot encoding text at word or character level. \n",
    "\n",
    "These built-in functions are to be preferred since they will take care of a number of important features, such as stripping special characters from strings or taking only the top N most common words in a data set, which is a common restriction to avoid dealing with very large input vector spaces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using Keras for one-hot encoding on word level works as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 9 unique tokens.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "\n",
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "\n",
    "# create a tokenizer, configured to only take the top N=1000 most common words\n",
    "tokenizer = Tokenizer(num_words=1000)\n",
    "# this builds the word index\n",
    "tokenizer.fit_on_texts(samples)\n",
    "\n",
    "# this turns strings into lists of integer indices\n",
    "sequences = tokenizer.texts_to_sequences(samples)\n",
    "\n",
    "# directly get the one-hot binary representation\n",
    "'''NOTE:''' \n",
    "# other vectorization modes than one-hot encoding are supported too !!!!\n",
    "one_hot_results = tokenizer.texts_to_matrix(samples, mode='binary') \n",
    "\n",
    "# recover the word index that was computed\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"ce33ff\">TASK 4</font><br>\n",
    "transfer what you have learned and implement a one-hot encoding on character level using Keras' built-in functions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add your code here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}