NLP Essentials.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLP Essentials\n",
    "\n",
    "## Libraries and Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the most powerful and useful NLP libraries in Python.\n",
    "\n",
    "•\t**Scikit-learn:** the most powerful and famous Machine Learning library in Python\n",
    "\n",
    "•\t**Natural Language Toolkit (NLTK):** The complete toolkit for all NLP techniques.\n",
    "\n",
    "•\t**TextBlob:** Easy to use NLP tools API, built on top of NLTK and Pattern.\n",
    "\n",
    "•\t**SpaCy:** Industrial strength NLP with Python and Cython.\n",
    "\n",
    "•\t**Gensim:** Topic Modelling for Humans\n",
    "\n",
    "•\t**Stanford Core NLP:** NLP services and packages by Stanford NLP Group.\n",
    "\n",
    "•\t**Fasttext:** NLP library for learning of word embeddings and sentence classification created by Facebook's AI Research (FAIR) lab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Knowing when to use which package all depends on the use case:\n",
    "\n",
    "- ***NLTK*** is recommended only as an education and research tool. Its modularized structure makes it excellent for learning and exploring NLP concepts, but it’s not meant for production. It’s the most famous Python NLP library, and it’s led to incredible breakthroughs in the field.  NLTK is also popular for education and research. On its own website, NLTK claims to be an “an amazing library to play with natural language.” The major drawback of NLTK is that it’s heavy and slippery, and it has a steep learning curve. The second major weakness is that it’s slow and not production-ready.\n",
    "\n",
    "- ***TextBlob*** is built on top of NLTK, and it’s more easily-accessible. This is our favorite library for fast prototyping or building applications that don’t require highly optimized performance. Beginners should start here. TextBlob makes text processing simple by providing an intuitive interface to NLTK. It’s a welcome addition to an already solid lineup of Python NLP libraries because it has a gentle learning curve while boasting a surprising amount of functionality.\n",
    "\n",
    "- ***Stanford’s CoreNLP*** is a Java library with Python wrappers. It’s in many existing production systems due to its speed. Stanford CoreNLP is a suite of production-ready natural analysis tools. It includes part-of-speech (POS) tagging, entity recognition, pattern learning, parsing, and much more. Many organizations use CoreNLP for production implementations. It’s fast, accurate, and able to support several major languages.\n",
    "\n",
    "- ***SpaCy*** is a new NLP library that’s designed to be fast, streamlined, and production-ready. It’s not as widely adopted, but if you’re building a new application, you should give it a try. SpaCy is minimal and opinionated, and it doesn’t flood you with options like NLTK does. Its philosophy is to only present one algorithm (the best one) for each purpose. You don’t have to make choices, and you can focus on being productive. Because it’s built on Cython, it’s also lightning-fast. \n",
    "\n",
    "- ***Gensim*** is most commonly used for topic modeling and similarity detection. It’s not a general-purpose NLP library, but for the tasks it does handle, it does them well. Gensim is a well-optimized library for topic modeling and document similarity analysis. Among the Python NLP libraries listed here, it’s the most specialized. Its topic modeling algorithms, such as its Latent Dirichlet Allocation (LDA) implementation, are best-in-class. In addition, it’s robust, efficient, and scalable.\n",
    "\n",
    "- ***Fasttext*** is mainly used for applying deep learning techniques to NLP problems. FastText combines some of the most successful concepts introduced by the natural language processing and machine learning communities in the last few decades. These include representing sentences with bag of words and bag of n-grams, as well as using subword information, and sharing information across classes through a hidden representation.  It also employs a hierachical softmax that takes advantage of the unbalanced distribution of the classes to speed up computation. These different concepts are being used for two different tasks: efficient text classification and learning word vector representations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Basic Text Processing and NLP Techniques\n",
    "\n",
    "The written text or spoken language is the most unstructured form of all the available data. Before we try to extract meaningful insights from text data, we need to sanitize the data by applying some text preprocessing techniques such as tokenization, lemmatization and stopword removal. In NLP, our first goal is to prepare the text data into a format that could be handled by machine learning algorithms. For this purpose, we're going to use the NLP packages mentioned above but let's start with NLTK package at first due to its simplicity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Installing NLTK package is a little bit different than installing other Python packages. NLTK package has many other sub modules and data that should be installed after we import NLTK.\n",
    "\n",
    "Once that we have confirmed that nltk is installed, we will have to download and install NLTK data. NLTK Data consists of the corpora and all the words in a language along with various grammar syntaxes, toy grammars, trained models, etc. They help the users to easily process languages by applying the various functions. Please note that installing nltk data is a long process it involves downloading over 1 GB of data. \n",
    "\n",
    "To download the nltk data, import NLTK and then run `nltk.download_shell()` or `nltk.download()` to see the download options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see the downloaded texts from Project Gutenberg. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['austen-emma.txt',\n",
       " 'austen-persuasion.txt',\n",
       " 'austen-sense.txt',\n",
       " 'bible-kjv.txt',\n",
       " 'blake-poems.txt',\n",
       " 'bryant-stories.txt',\n",
       " 'burgess-busterbrown.txt',\n",
       " 'carroll-alice.txt',\n",
       " 'chesterton-ball.txt',\n",
       " 'chesterton-brown.txt',\n",
       " 'chesterton-thursday.txt',\n",
       " 'edgeworth-parents.txt',\n",
       " 'melville-moby_dick.txt',\n",
       " 'milton-paradise.txt',\n",
       " 'shakespeare-caesar.txt',\n",
       " 'shakespeare-hamlet.txt',\n",
       " 'shakespeare-macbeth.txt',\n",
       " 'whitman-leaves.txt']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.corpus.gutenberg.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "burgess = nltk.corpus.gutenberg.raw('burgess-busterbrown.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we loaded a whole text file into our project. Since the size of this text is quite large, we can split that by paragraph and select a few of them or just select as many words as we want by splitting into words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[The Adventures of Buster Bear by Thornton W. Burgess 1920]\r\n",
      "\r\n",
      "I\r\n",
      "\r\n",
      "BUSTER BEAR GOES FISHING\r\n",
      "\r\n",
      "\r\n",
      "Buster Bear yawned as he lay on his comfortable bed of leaves and\r\n",
      "watched the first early morning sunbeams creeping through the Green\r\n",
      "Forest to chase out the Black Shadows. Once more he yawned, and slowly\r\n",
      "got to his feet and shook himself. Then he walked over to a big\r\n",
      "pine-tree, stood up on his hi\n"
     ]
    }
   ],
   "source": [
    "# here, we just print the first 200 letters from the corpus\n",
    "print (burgess[:400])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenization is a step which splits longer strings of text into smaller pieces, or tokens. Larger chunks of text can be tokenized into sentences, sentences can be tokenized into words, etc. This immediately turns an unstructured string (text document) into a more usable data, which can be further structured, and made more suitable for machine learning. Further processing is generally performed after a piece of text has been appropriately tokenized. There are three different ways i.e Split, RegexpTokenizer and Word Tokenize.\n",
    "\n",
    "Let's tokenize this text by using pure python at first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert all letters to lowercase in order to standardize the text.\n",
    "burgess = burgess.lower()\n",
    "\n",
    "# here, we just split the whole text by spaces \n",
    "tokens = [word for word in burgess.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[the', 'adventures', 'of', 'buster', 'bear', 'by', 'thornton', 'w.', 'burgess', '1920]', 'i', 'buster', 'bear', 'goes', 'fishing', 'buster', 'bear', 'yawned', 'as', 'he', 'lay', 'on', 'his', 'comfortable', 'bed', 'of', 'leaves', 'and', 'watched', 'the', 'first', 'early', 'morning', 'sunbeams', 'creeping', 'through', 'the', 'green', 'forest', 'to', 'chase', 'out', 'the', 'black', 'shadows.', 'once', 'more', 'he', 'yawned,', 'and', 'slowly', 'got', 'to', 'his', 'feet', 'and', 'shook', 'himself.', 'then', 'he', 'walked', 'over', 'to', 'a', 'big', 'pine-tree,', 'stood', 'up', 'on', 'his', 'hind', 'legs,', 'reached', 'as', 'high', 'up', 'on', 'the', 'trunk', 'of', 'the', 'tree', 'as', 'he', 'could,', 'and', 'scratched', 'the', 'bark', 'with', 'his', 'great', 'claws.', 'after', 'that', 'he', 'yawned', 'until', 'it', 'seemed']\n"
     ]
    }
   ],
   "source": [
    "print(tokens[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not the best way as we can see there are some word with symbols like `[the`. Now let's do the same with NLTK tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[', 'the', 'adventures', 'of', 'buster', 'bear', 'by', 'thornton', 'w.', 'burgess', '1920', ']', 'i', 'buster', 'bear', 'goes', 'fishing', 'buster', 'bear', 'yawned', 'as', 'he', 'lay', 'on', 'his', 'comfortable', 'bed', 'of', 'leaves', 'and', 'watched', 'the', 'first', 'early', 'morning', 'sunbeams', 'creeping', 'through', 'the', 'green', 'forest', 'to', 'chase', 'out', 'the', 'black', 'shadows', '.', 'once', 'more', 'he', 'yawned', ',', 'and', 'slowly', 'got', 'to', 'his', 'feet', 'and', 'shook', 'himself', '.', 'then', 'he', 'walked', 'over', 'to', 'a', 'big', 'pine-tree', ',', 'stood', 'up', 'on', 'his', 'hind', 'legs', ',', 'reached', 'as', 'high', 'up', 'on', 'the', 'trunk', 'of', 'the', 'tree', 'as', 'he', 'could', ',', 'and', 'scratched', 'the', 'bark', 'with', 'his', 'great']\n"
     ]
    }
   ],
   "source": [
    "tokens = nltk.word_tokenize(burgess)\n",
    "print(tokens[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also tokenizee the text with custom regex patterns which is called RegexpTokenizer. Using this function, we can apply highly customized and detailed tokenization that could help us extract much more information without further processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the', 'adventures', 'of', 'buster', 'bear', 'by', 'thornton', 'w', 'burgess', '1920', 'i', 'buster', 'bear', 'goes', 'fishing', 'buster', 'bear', 'yawned', 'as', 'he', 'lay', 'on', 'his', 'comfortable', 'bed', 'of', 'leaves', 'and', 'watched', 'the', 'first', 'early', 'morning', 'sunbeams', 'creeping', 'through', 'the', 'green', 'forest', 'to', 'chase', 'out', 'the', 'black', 'shadows', 'once', 'more', 'he', 'yawned', 'and', 'slowly', 'got', 'to', 'his', 'feet', 'and', 'shook', 'himself', 'then', 'he', 'walked', 'over', 'to', 'a', 'big', 'pine', 'tree', 'stood', 'up', 'on', 'his', 'hind', 'legs', 'reached', 'as', 'high', 'up', 'on', 'the', 'trunk', 'of', 'the', 'tree', 'as', 'he', 'could', 'and', 'scratched', 'the', 'bark', 'with', 'his', 'great', 'claws', 'after', 'that', 'he', 'yawned', 'until', 'it']\n"
     ]
    }
   ],
   "source": [
    "tokenizer = nltk.tokenize.RegexpTokenizer(r'\\w+')\n",
    "tokens = tokenizer.tokenize(burgess)\n",
    "print(tokens[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you see, wee don't see any punctuations since we tokenized by words in regex. \n",
    "\n",
    "Let's see the frequeencies of each token. This is quite simple and straightforward with NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_freq = nltk.FreqDist(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This return a dictionary like key-value pair and in order to find a frequency of a token, we simply use that token as key within brackets just as we do in dictionary types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_freq[\"bark\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and we can easily plot this with an embedded plot function..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEgCAYAAACuDOSlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8FfW9//HXJzsJCTsh7PsmoJKAIq6Ia23VtmqtrWi19lbb29bbqv21vW3v494utrfW2lv3BW1ra20toqgFRBBkMVEEZN9BIOwhkBCyfH5/zAQDhuRkOTlZ3s/HYx7nzJz5zHxOlvM53+/MfMfcHRERkZPFxToBERFpnlQgRESkWioQIiJSLRUIERGplgqEiIhUSwVCRESqpQIhIiLVilqBMLNhZra0ynTIzL5tZp3NbKaZrQsfO4Xrm5n9zszWm9kyMxsbrdxERKR2USsQ7r7G3c9w9zOAbKAIeAm4D5jt7kOA2eE8wBXAkHC6A3g4WrmJiEjtEppoPxcDG9x9i5ldDVwYLp8KvAXcC1wNPOvBpd2LzKyjmWW5+85TbbRr167ev3//eiVUXFxMu3bt6hWreMXHOr455KD4lhufl5e319271bqiu0d9Ap4CvhE+P3jSawfCx1eAc6ssnw3k1LTd7Oxsr6/c3Nx6xype8bGObw45KL7lxgO5HsFnt3mUx2IysyRgB3Cau+eb2UF371jl9QPu3snMXgV+7u7zw+WzgXvcPe+k7d1B0AVFVlZW9vTp0+uVV1FREampqfV7U4pXfIzjm0MOim+58Tk5OXnunlPripFUkYZMBF1H/6oyvwbICp9nAWvC548CN1a33qkmtSAU31bjm0MOim+58UTYgmiK01xvBJ6vMv8yMCV8PgWYVmX5zeHZTGcDBV7D8QcREYmuqB6kNrNU4BLga1UW/wJ4wcxuA7YC14XLZwBXAusJzni6NZq5iYhIzaJaINy9COhy0rJ9BGc1nbyuA3dFMx8REYmcrqQWEZFqtckCUVGhu+iJiNSmTRaIP7y1nh/O2cery3ZSWl4R63RERJqlprqSutlwd6Yt3cG6vaXc9ef3yMxI5qaz+vGF8X3onp4S6/RERJqNNteCMDP+cec53H5mOoO6pZF/qITfzFzLxF+8ybf+8j55Ww5UXochItKmtbkWBEB6SiJXDE7jB9eP5Z0N+5j6zmZmrcpn2tIdTFu6g1G9Mrh5Qn8+c3pPUhLjY52uiEhMtMkCUcnMmDi4KxMHd2X7gSL+tHgrf1mylRUfHeKeF5fxsxmruGFcH750Vj/6dG7YsAgiIi1Nm+tiOpXenVK59/LhLPz+xfz6utMZ3asDB4tKeXTuRs7/1Rxun5rL2+v2qPtJRNqMNt2CqE5KYjyfz+7N58b2Yum2gzy7cAuvLNvBrFX5zFqVz8BuaUyZ0J+BcTr7SURaNxWIUzAzzuzbiTP7duL/XTmCvyzZyp8Wb2XjniP8+OUPyUgyXh1apK4nEWm11MUUgW7pyXzz4iG8fe9F/OGmsQzvkc6hY86LedtjnZqISNSoQNRBYnwcV47O4t7LhwMwZ83uGGckIhI9KhD1MGFQF5LiYdn2AnYfOhrrdEREokIFoh5SEuMZ3T0ZUCtCRFovFYh6yskKCsTsVSoQItI6qUDUU3ZYIOav30tJWXmMsxERaXwqEPXUJTWeEVkZFB0rZ/HG/bFOR0Sk0alANMDFw7sD8OZqdTOJSOujAtEAk0YEBWL26nwNwSEirY4KRAOc3rsjXdKS2La/mPW7D8c6HRGRRqUC0QDxccYFw7oB6mYSkdZHBaKBLh6eCcBsFQgRaWVUIBrovKFdSYgz8rYc4GDRsVinIyLSaFQgGigjJZHxAzpTXuHMXbsn1umIiDQaFYhGMEmnu4pIK6QC0QgqC8TctXsoK9eNhESkdVCBaAQDu7VnQNc0DhaV8v62g7FOR0SkUahANJLKVoQG7xOR1kIFopF8POxGfowzERFpHFEtEGbW0cxeNLPVZrbKzCaYWWczm2lm68LHTuG6Zma/M7P1ZrbMzMZGM7fGltO/M+2TE1ibf5ht+4tinY6ISINFuwXxIPC6uw8HTgdWAfcBs919CDA7nAe4AhgSTncAD0c5t0aVlBDH+UO7ArqJkIi0DlErEGaWAZwPPAng7sfc/SBwNTA1XG0qcE34/GrgWQ8sAjqaWVa08ouGSZVXVes4hIi0AtFsQQwE9gBPm9n7ZvaEmaUBme6+EyB87B6u3wvYViV+e7isxbhwWDfMYOGGfRwpKYt1OiIiDWLRGqbazHKARcBEd19sZg8Ch4BvunvHKusdcPdOZvYq8HN3nx8unw3c4+55J233DoIuKLKysrKnT59er/yKiopITU2tV2xN8d+fvY+1+0u595yOjO+V0uT7V3zbiG8OOSi+5cbn5OTkuXtOrSu6e1QmoAewucr8ecCrwBogK1yWBawJnz8K3Fhl/ePrnWrKzs72+srNza13bE3xv5u11vvd+4rf9/cPYrJ/xbeN+OaQg+JbbjyQ6xF8jketi8nddwHbzGxYuOhiYCXwMjAlXDYFmBY+fxm4OTyb6WygwMOuqJbk+E2EVu3WTYREpEVLiPL2vwn8ycySgI3ArQTHPV4ws9uArcB14bozgCuB9UBRuG6LMzIrgx4ZKew6dJQPdxxiVK8OsU5JRKReolog3H0pUF0/18XVrOvAXdHMpymYGZNGdOfPi7cye9VuFQgRabF0JXUUTBqmq6pFpOVTgYiCiYO7kpwQxwfbC9hTWBLrdERE6kUFIgraJcVzzqAugK6qFpGWSwUiSiaNCK6qflNXVYtIC6UCESWVw3+/vW4PJWXlMc5GRKTuVCCipFfHdgzvkc6RY+Us2bQ/1umIiNSZCkQU6V7VItKSqUBE0cW6qlpEWjAViCg6o08nOqUmsnV/ERv2HIl1OiIidaICEUXxccaFumhORFooFYgo03EIEWmpVCCi7Pyh3YiPM97dfICC4tJYpyMiEjEViCjr0C6Rcf07UV7hzFu7J9bpiIhETAWiCaibSURaIhWIJjBpeDDsxltrdlNeodNdRaRlUIFoAoO6pdGvSyoHikpZuu1ArNMREYmICkQTMLPj3UyzNXifiLQQKhBN5OKwm0nHIUSkpVCBaCLjB3QmLSme1bsK2X6gKNbpiIjUSgWiiSQlxHHekG4AzFErQkRaABWIJjRphE53FZGWQwWiCV0Ujsu0YMM+jpZVxDgbEZGaqUA0oW7pyZzeuwPHyipYvvtYrNMREamRCkQTq7xoLm9nSYwzERGpmQpEE6u8iVDezhLdREhEmjUViCZ2Ws8MMjOS2V9cwYqPDsU6HRGRU1KBaGJmxuWn9QDgkbkbYpyNiMipqUDEwL9dOIikOHh1+U6WbT8Y63RERKoV1QJhZpvNbLmZLTWz3HBZZzObaWbrwsdO4XIzs9+Z2XozW2ZmY6OZWyxldWjHFUPSALj/9TUxzkZEpHpN0YK4yN3PcPeccP4+YLa7DwFmh/MAVwBDwukO4OEmyC1mrh2eRnpKAvPX72X+ur2xTkdE5BNi0cV0NTA1fD4VuKbK8mc9sAjoaGZZMcivSaQnxfFvFwwC4P43VuuMJhFpdqJdIBz4l5nlmdkd4bJMd98JED52D5f3ArZVid0eLmu1vjJxAN3Tk1m2vYDXVuyKdToiIiewaH5zNbOe7r7DzLoDM4FvAi+7e8cq6xxw905m9irwc3efHy6fDdzj7nknbfMOgi4osrKysqdPn16v3IqKikhNTa1XbGPGv7GhiMfeO0TP9vH89rKuxMdZk+5f8S0zvjnkoPiWG5+Tk5NXpdv/1Ny9SSbgJ8B3gTVAVrgsC1gTPn8UuLHK+sfXO9WUnZ3t9ZWbm1vv2MaMP1ZW7hfc/6b3u/cV//PiLU2+f8W3zPjmkIPiW248kOsRfG5HrYvJzNLMLL3yOXApsAJ4GZgSrjYFmBY+fxm4OTyb6WygwMOuqNYsMT6O/7h0GAC/nbWWo6XlMc5IRCQQzWMQmcB8M/sAWAK86u6vA78ALjGzdcAl4TzADGAjsB54HLgzirk1K58ancVpPTPIP1TCM+9sjnU6IiIAJERrw+6+ETi9muX7gIurWe7AXdHKpzmLizPuvXw4Nz+1hD/MWc+N4/rSITUx1mmJSBunK6mbifOGdGXCwC4cOlrGI/M0BIeIxJ4KRDNhZtxzeXAs4ukFm8g/dDTGGYlIW6cC0Yyc2bcTl5/Wg6OlFfxu9rpYpyMibZwKRDPz3cuGEmfwl3e3sWnvkVinIyJtmApEMzO4ezrXZfehvML5339pID8RiR0ViGboW5OHkJQQxyvLdrJ8e0Gs0xGRNkoFohnq2bEdUyb0A4KB/EREYkEFopm688LBpCcn8Pa6vbyzXsOBi0jTU4FopjqlJfG1CwYC8MvXNRy4iDQ9FYhm7CvnDqBr+2Q+2F7AGx9qOHARaVoqEM1YalIC37p4MAD3v7GGsvKKGGckIm2JCkQzd8O4vvTtnMrGPUf4+3vbY52OiLQhdS4QZtbJzMZEIxn5pKSEOP7j0qEAPDBznYYDF5EmE1GBMLO3zCzDzDoDHwBPm9lvopuaVPr0mJ6MzMpg16GjPLtwc6zTEZE2ItIWRAd3PwR8Fnja3bOBydFLS6qKi/t4IL//m7OBguLSGGckIm1BpAUiwcyygOuBV6KYj5zCBUO7cdaAzhQUl/KYhgMXkSYQaYH4KfAGsN7d3zWzgYCGG21CZsa9VwwH4Kn5mzlQrGMRIhJdkRaIne4+xt3vhON3i9MxiCY2tm8nLh2ZSXFpOX9bdTjW6YhIKxdpgXgowmUSZd+7bBhxBrM2FrNln4YDF5HoqfGe1GY2ATgH6GZmd1d5KQOIj2ZiUr0hmelce2Zv/v7edh6bt5H/uXZ0rFMSkVaqthZEEtCeoJCkV5kOAZ+PbmpyKl+/MBij6cW87ew7XBLjbESktaqxBeHuc4G5ZvaMu29popykFoO7p5OdlUzezhKeW7SFb08eGuuURKQVivQYRLKZPWZm/zKzNyunqGYmNbp6WBoAzy7coqurRSQqamxBVPE34BHgCUCfRs3AyK6JjOndgWXbC/j7e9u56ax+sU5JRFqZSFsQZe7+sLsvcfe8yimqmUmNzIyvnhcci3ji7U1UVOh+ESLSuCItENPN7E4zyzKzzpVTVDOTWl0xqge9OrZj094jzFqVH+t0RKSVibRATAG+B7wD5IVTbrSSksgkxMdx27kDAHj87Y0xzkZEWpuICoS7D6hmGhjt5KR214/rQ0ZKAu9uPsB7Ww/EOh0RaUUiHe775uqmaCcntWufnMBNZwcHqB+fp1aEiDSeSLuYxlWZzgN+AnwmkkAzizez983slXB+gJktNrN1ZvZXM0sKlyeH8+vD1/vX8b20Wbec05/EeOP1D3dp+A0RaTSRdjF9s8r0VeBMgqusI/EtYFWV+V8CD7j7EOAAcFu4/DbggLsPBh4I15MIZGakcM0ZvXCHJ+dvinU6ItJK1Pee1EXAkNpWMrPewKcIrp/AzAyYBLwYrjIVuCZ8fnU4T/j6xeH6EoGvnh8cEnohdxsHjhyLcTYi0hpEegxiupm9HE6vAmuAaRGE/ha4B6gI57sAB929LJzfDvQKn/cCtgGErxeE60sEhmamc+GwbhwtreCPizQqiog0nLnXfoGVmV1QZbYM2OLu22uJuQq40t3vNLMLge8CtwILw24kzKwPMMPdR5vZh8Bllds1sw3AeHffd9J27wDuAMjKysqePn16ZO/0JEVFRaSmptYrtrnGL99dwk/mHqBDchyPfKobSfGnboA1x/wV37JyUHzLjc/Jyclz95xaV3T3iCYgE7gqnLpHsP7PCVoIm4FdBN1SfwL2AgnhOhOAN8LnbwATwucJ4XpW0z6ys7O9vnJzc+sd21zjKyoq/MoH53m/e1/x5xdvafL9K77p4ptDDopvufFArkfwuR9pF9P1wBLgOoL7Ui82sxqH+3b377t7b3fvD3wBeNPdbwLm8PFQ4VP4uKvq5XCe8PU3wzciETIz7giPRTz+9kYNvyEiDRLpQeofAOPcfYq73wyMB35Uz33eC9xtZusJjjE8GS5/EugSLr8buK+e22/TrhydRc8OKWzYc4Q5a3bHOh0RacEiLRBx7l7102ZfHWJx97fc/arw+UZ3H+/ug939OncvCZcfDecHh6/rqq96SIyP4yvh8BuP6cI5EWmASD/kXzezN8zsFjO7BXgVmBG9tKQhbhjXh/TkBBZv2s8H2w7GOh0RaaFqLBBmNtjMJrr794BHgTHA6cBC4LEmyE/qIT0lkS+e3RfQIH4iUn+1tSB+CxQCuPs/3P1ud/8OQevht9FOTurv1nMGkBBnzFi+k237i2Kdjoi0QLUViP7uvuzkhe6eC/SPSkbSKHp0SOEzZ/SkwuGpBRp+Q0TqrrYCkVLDa+0aMxFpfJV3nPvru9soKCqNcTYi0tLUViDeNbOvnrzQzG4juGmQNGMjsjI4b0hXio6V86clGn5DROomoZbXvw28ZGY38XFByCEYyfXaaCYmjeOO8wfy9rq9PL1gM7edO4DkhPhYpyQiLUSNLQh3z3f3c4CfEgyZsRn4qbtPcPdd0U9PGurcwV0Z3iOdPYUlTFu6I9bpiEgLEun9IOa4+0Ph9Ga0k5LGc8LwG/M2otFLRCRS9b0fhLQgV43pSY+MFNbtPsxba/fEOh0RaSFUINqApIQ4bp3YH9B9q0UkcioQbcSNZ/WlfXIC72zYx4qPCmKdjoi0ACoQbURGSiI3ju8DaPgNEYmMCkQbcuvEYPiNV5btZE9ReazTEZFmTgWiDenZsR1XjcmivMJ5dd2RWKcjIs2cCkQbc3s4/MbMjcUafkNEaqQC0caM6tWBiYO7cLTMueGxhWzZp5aEiFRPBaIN+tm1o+mZHs/qXYV85vcLmKtrI0SkGioQbVC/Lmn84uIuTB6RSUFxKbc8vYQ/vLVeV1mLyAlUINqotMQ4HvtyNt+ZPBR3uP/1Ndz15/c4UlIW69REpJlQgWjD4uKMb00ewhM355CenMCM5bu49g8L2LRXxyVERAVCgMkjM/nnNyYyqFsaa/MP85nfz2fO6t2xTktEYkwFQgAY1K09/7xrIpeOzKTwaBlfmfouv39zHRUVOi4h0lapQMhx6SmJPPKlbL576VAAfv2vtXz9T3kc1nEJkTZJBUJOEBdnfGPSEJ6aMo70lATe+DCfa/5vARv2HI51aiLSxFQgpFoXDe/Oy984lyHd27N+92Gu+f0CZq3Mj3VaItKEVCDklAZ0TeOluyZyxageFJaUcfuzuTw4S8clRNoKFQipUfvkBP5w01juuXwYZvDArLV87Y95FJVWxDo1EYmyhFgnIM2fmXHnhYMZmZXBvz//PjNX5rN8azyfK1jNyKwOjOyZQb/OqcTFWaxTFZFGFLUCYWYpwDwgOdzPi+7+YzMbAPwF6Ay8B3zZ3Y+ZWTLwLJAN7ANucPfN0cpP6u7CYd2Z/s1z+dpzeazeVcj/zdlw/LW0pHhGZGUwsmcGI8PHoZnppCTGxzBjEWmIaLYgSoBJ7n7YzBKB+Wb2GnA38IC7/8XMHgFuAx4OHw+4+2Az+wLwS+CGKOYn9dCvSxr/vGsiT8xYxNGUrqzceYiVOw6x69BRcrccIHfLgePrxscZg7qlHS8Yla2NzmlJMXwHIhKpqBUID0Z+qzw3MjGcHJgEfDFcPhX4CUGBuDp8DvAi8HszM9cIcs1OSmI8E3qnkJ097PiyfYdLWLWzkJU7C1i54xArdx5iw54jrM0/zNr8w/xz6Y7j6/bISKFXmnP1sc1cPCKTXh3bxeJtiEgtLJqfv2YWD+QBg4H/A34FLHL3weHrfYDX3H2Uma0ALnf37eFrG4Cz3H3vSdu8A7gDICsrK3v69On1yq2oqIjU1NT6vTHFRxRfUu5sKyhj88FSNh0MHjcXlHG07MS/uQEdE8jpmcy4nikM7JiAWe3HMlrC+49mfHPIQfEtNz4nJyfP3XNqXdHdoz4BHYE5wHnA+irL+wDLw+cfAr2rvLYB6FLTdrOzs72+cnNz6x2r+PrHl5dX+KY9h/3+F+f5157N9RE/es373fvK8ems/5nlP3hpmc9Zne9HS8saff+tJb455KD4lhsP5HoEn91NchaTux80s7eAs4GOZpbg7mVAb6Cy72F7WDC2m1kC0AHY3xT5SdOJizP6d01jUv9UvpedzdHSchZu3MeslfnMWpXPrkNH+eOirfxx0VbSkuI5f2g3Jo/I5KLh3XXsQqSJRfMspm5AaVgc2gGTCQ48zwE+T3Am0xRgWhjycji/MHz9zbDSSSuWkhjPRcO6c9Gw7vz3NaNY8dEhZq7KZ9bKfFbuPMRrK3bx2opdxBnk9OvM5JHdmTwiM9Zpi7QJ0WxBZAFTw+MQccAL7v6Kma0E/mJm/w28DzwZrv8k8JyZrSdoOXwhirlJM2RmjO7dgdG9O3D3JUPZfqCIN1fvZubKfBZt3MeSzftZsnk/P5uxmp7p8Xw6fxWXjMzkzL6diNc1GCKNLppnMS0Dzqxm+UZgfDXLjwLXRSsfaXl6d0rl5gn9uXlCfw4dLWXe2j3MWpnPm6t3s6OwjEfnbeTReRvpnJbEpOFBy+K8IV1JS9b1nyKNQf9J0iJkpCRy1ZieXDWmJ6XlFfz5X4vYWt6RmSvz2bq/iBfztvNi3naSEuKYOKgLk0dmMnlEJpkZKbFOXaTFUoGQFicxPo5R3ZOZkj2SH35qBOt3Hz5+3OL9bQeZs2YPc9bs4QcvrWBM7w5cMiKTySMzGd4jPaJTaEUkoAIhLZqZMSQznSGZ6dx54WB2Fx5lzurdzFy5m/nr97BsewHLthfwvzPX0qtjOy4JWxYJGpFWpFYqENKqdE9P4YZxfblhXF+Kj5WzYP1eZq3KZ9aq3Xx0sJhn3tnMM+9spn2i8du0fCaP1BlRIqeiAiGtVruk+OBYxMhMKiqcD7YfZNaqfGauzGdt/mHu/NN7PHXLOM4d0jXWqYo0S7ofhLQJcXHGmX078b3LhvPGt8/n8kGpHCuv4KvP5pK3RddjilRHBULaHDPjtjPT+dzY3hSXlnPL0++y4qOCWKcl0uyoQEibFGfGLz83Orid6tEybn5qCet3F8Y6LZFmRQVC2qyE+Dge/MKZXDC0G/uPHOOmJxazbX9RrNMSaTZUIKRNS0qI45EvZTN+QGfyD5XwxScWsavgaKzTEmkWVCCkzWuXFM+TU3I4vXcHtu0v5ktPLmbf4ZJYpyUScyoQIkB6SiLP3DqeYZnprN99mJufWkJBcWms0xKJKRUIkVCntCSeu308/buk8uGOQ3zlmXcpOlYW67REYkYFQqSK7ukp/PH2s+jZIYW8LQe449k8jpaWxzotkZhQgRA5Se9Oqfzx9rPo2j6Z+ev38s3n36e0vCLWaYk0ORUIkWoM7Nae524bT4d2icxcmc93//YB5RrgT9oYFQiRUxiRlcHUr4wnLSmeaUt38MN/rkB3wZW2RAVCpAZn9OnIE1PGkZwQx/NLtvKzGatUJKTNUIEQqcWEQV145EvZJMYbj7+9id/NXh/rlESahAqESAQuGt6d395wJnEGD8xay/S1R2KdkkjUqUCIROhTY7L45efGAPDMB4V8/Y95LNq4T11O0mrphkEidXBdTh+OllXwk2kreG3FLl5bsYthmencfE4/rjmjF2nJ+peS1kN/zSJ19OWz+9GjLJ8VxR3585KtrMkv5AcvreAXr63muuw+fHlCPwZ0TYt1miINpi4mkXro3C6e71wylAX3TuLBL5xBdr9OFB4t46kFm7jo128x5aklvLk6nwpdOyEtmFoQIg2QlBDH1Wf04uozerHiowKeXbiZaUt3MHftHuau3UPfzql8+ex+XJ/Thw6pibFOV6RO1IIQaSSjenXg/s+fzqLvX8z3rxhO707t2Lq/iP+ZsYqzfj6L7/9jGSt3HIp1miIRUwtCpJF1SkviaxcM4vbzBjJn9W6mLtzM2+v28vySbTy/ZBvj+3fmnMxy0nsX0q19Mh1TEzGzWKct8gkqECJREh9nTB6ZyeSRmWzYc5jnFm7hxbztLNm8nyWb4beL5wGQGG90bZ9Mt/RkulU+plc/n5qkf1lpOlH7azOzPsCzQA+gAnjM3R80s87AX4H+wGbgenc/YMFXqAeBK4Ei4BZ3fy9a+Yk0pUHd2vOTz5zGdy8bxkvvf8QL76ylyBPZU1jCoaNl7Cw4ys4IbnWalhRPt/RkenRI4ayu5Zx5phMXp9aHREc0v46UAf/h7u+ZWTqQZ2YzgVuA2e7+CzO7D7gPuBe4AhgSTmcBD4ePIq1G++QEvnx2P0Ym7iU7OxuAo6Xl7D1cwp7CcKr6PJzffSh4PHKsnCP7iti8r4hFG2Hmtvncd8VwzhvSVd1U0uiiViDcfSewM3xeaGargF7A1cCF4WpTgbcICsTVwLMeXJa6yMw6mllWuB2RVislMZ7enVLp3Sm1xvXcnUNHy9hTWELu5v386rUPWbnzEDc/tYSJg7tw3+UjGN27QxNlLW1Bk3Romll/4ExgMZBZ+aHv7jvNrHu4Wi9gW5Ww7eEyFQgRwMzo0C6RDu0SGdy9PX3ZzbLiTvxhznoWrN/Hp38/n6vGZPHdS4fRXxfqSSOwaI8jY2btgbnA/7j7P8zsoLt3rPL6AXfvZGavAj939/nh8tnAPe6ed9L27gDuAMjKysqePn16vfIqKioiNbXmb2yKV3xzja+6jcJjFby0+ggz1h2htALiDS4ZmMr1I9PokBIftRwU33Ljc3Jy8tw9p9YV3T1qE5AIvAHcXWXZGiArfJ4FrAmfPwrcWN16p5qys7O9vnJzc+sdq3jFxzq+um18dKDIv/vCUh9w3yve795XfOSPXvMHZq7xwqOlUclB8S03Hsj1CD7Do3ahXHhW0pPAKnf/TZWXXgamhM+nANOqLL/ZAmcDBa7jDyIR69mxHb+67nRe+9b5TB7RnSPHyvntrHVc+Ks5PLtwM8fKdF9tqZtoXkk9EfgyMMnMlobTlcAvgEvMbB1wSTgPMAPYCKwHHgfujGJuIq3WsB7pPDFlHC98bQJj+3Zk7+Fj/Oe0D7nkgbnDLPpaAAATfUlEQVRM/2CHxoeSiEXzLKb5wKnOu7u4mvUduCta+Yi0NeMHdObvXz+HNz7M5/43VrNxzxG++fz7PP72Ru69fDjJuo+F1EKXZYq0YmbG5aN6MHlEd/6Wt50HZq5l2fYCbnpiMb3S4/ns/jVcMSqLEVnpuo5CPkEFQqQNSIiP48bxfbnmjF48tWATT87fxEeFx3jozfU89OZ6+ndJ5YrRWVw5KotRvTJULARQgRBpU9olxXPXRYP52vkDee6NRawvSeeND3exeV8RD7+1gYff2kDvTu24cnQWV4zqwRl9OqpYtGEqECJtUEJ8HGMyk7k1ezT/dfUo3t28n9eW7+S1FbvYfqCYx+Zt5LF5G+nZIYXLR2Vx5egejO3bSeM+tTEqECJtXHyccfbALpw9sAs//vRp5G09wIzlO3l9xS52FBzlqQWbeGrBJrqnJ3PFqB5cMTqLcf07xzptaQIqECJyXFycMa5/Z8b178yPPjWSpdsP8tryncxYvouPDhYzdeEWpi7cQpe0JPqlGzn5qxjcvT1DurdncPf2pKfornmtiQqEiFQrLs4Y27cTY/t24v9dOYLlHxUwY/kuZizfydb9Rew7Au/t2nhCTFaHlLBgpDMkMygcQ7qn63arLZQKhIjUyswY07sjY3p35N7Lh7Fx7xFee+cDKtp3Z93uw6zLL2TjniPH72vx9rq9J8R3S09mSPf2DM1MZ3D39gzq1p6dBaV0319EalI8ackJJCfE6YB4M6MCISJ1YmYM6taeCb1TyM4ecnx5WXkF2w4Usy6/kHW7D7N+92HW7S5k/e7Dx+9t8c6GfSdu7F9zjj+NM0hLSiA1Of74Y2pSAmlJ8aQmh49JCaSFy/fnH2GDbzsxJiw2lTGpifE6sN4AKhAi0igS4uMY0DWNAV3TuPS0j5dXVDgfHSxm3e5C1uUfZt3uw2zae4Q9BwqpiE+k6Fg5R0rKKCmroLCkjMKSMqAksp0uXVbrKu0S448XlarFo7y4kIHbVtAxNRhCvWNqEh3bJdIxNTFclkSHdokkJURzRKLmTQVCRKIqLs7o0zmVPp1TmTQ88/jyvLy843fVg6AFUlRaTlFJOUeOlX38eKyMIyXlJzweLilny0c7SevQ+ROvH6kyX3SsnOLSYIJjn8htwbYtteaflhRPx9SksIgkHi8o+/cV0G3b8nr9TAyjuKCQD49tbtb3HW8+mYhIm5YQH0dGfBwZEZ4JlZd3hOzs02tcp6LCKS49ueCUc7ikjA9WrqVTZm8OFpVysPgYBUWlHCwu5WDRMQ4Wlx6fP3KsnCPHivnoYPEnd7Bha33e6nEvrvrwE8sq7zt+fKpSQLqnpxx/XtYEgy6qQIhIqxUXZ0G3UnICpJ/4WsbhbWRn968x3t05XFLGwaJSCopLPy4mxaVs3ryVvn371CuvCoeV67eQkN75hPuQ7y488b7jNcnOSubv4+q1+4ipQIiInIKZkZ6SSHpKIieXgryEvbUWmJrkJe8jO3v0CcvcncKSsuMH9Y9Phz8536Vd9I+NqECIiDQTZkZGSiIZKYkM6ta+xnVzc3Ojnk/bPTwvItKCNcU1IyoQIiJSLRUIERGplgqEiIhUSwVCRESqpQIhIiLVUoEQEZFqqUCIiEi1zD3643lEi5ntAWofbat6XYG9ta6leMU3z/jmkIPiW258P3fvVuta7t4mJyBX8YpvqfHNIQfFt+z4SCZ1MYmISLVUIEREpFptuUA8pnjFt+D45pCD4lt2fK1a9EFqERGJnrbcghARkRqoQIiISLVUICRiZpYcybIo59DJzMab2fmVU1Puvy0zs19Gsqy5MrPrzCw9fP5DM/uHmY2NdV7NmQpEC2NmmWZ2VTh1jzDmufDxWw3c/cIIl50qj0/svy45mdntwDzgDeCn4eNPIo1vCDOLM7MVTbGvZuySapZd0dRJmNk5ZvZFM7u5coow9EfuXmhm5wKXAVOBh6OX6SeZ2UQzSwuff8nMfmNm/SKMjTezX0U3wxO1qQJhZkPNbHblP7qZjTGzH9YhPtPMnjSz18L5kWZ2WwRxhWZ26FRTHfZ/PbAEuA64HlhsZp+PIDQ7/CP8SvgNvHPVKYL99jCzbKCdmZ1pZmPD6UIgNdL8gSnVLLulDvHfAsYBW9z9IuBMYE+kwWZ2v5llmFli+Hew18y+FEmsu1cAH5hZ3zrkW7nfRvn9h9uq999wfd+/mX3dzJYDw8xsWZVpE7CsDrl/K9y/hf9H75nZpZHGh9t4Dvg1cC7B38I4ICfC8PLw8VPAw+4+DUiKcL/zw8eTf5eFdfwdPgwUmdnpwD0EI0E8G0mgu5cT/C9H/1ZyVXbaZiZgLjAeeL/KshV1iH+N4IP5g3A+AVheh/j/Au4E0oEM4OvAPXWI/wDoXmW+W2UutcT9O7AKKAE2Vpk2ARsjiJ8CzAEKw8fK6WXgsxHE3whMBw6EMZXTHGBWHd7/u+HjUiC58nkd4peGj9cSfHvsHMnPr0r8m+HPYHbV99FUv/+G/g3X9/0DHYD+wPNAvypT5zrmXvl/c1n4szsdeK+O21hFePZlXSfgFeBRYAPQEUiuy++/MabK9wv8J3Bb1WURxv9v+LP7MvDZyila+SbQtqS6+5KTCnBZHeK7uvsLZvZ9AHcvM7Py2oKquMzdz6oy/7CZLQbujzA+zt13V5nfRwStQHf/HfA7M3sYeASo7Lef5+4fRBA/FZhqZp9z979HmGtV7wA7CcaO+d8qywupwzdQYLuZdQT+Ccw0swPAjjrEJ4aPVwLPu/v+On4Z+2ldVq5GQ3//0LC/4Xq9f3cvAAoICn1DVO7sSuBpd/+gHt+GVwA9CP6e6up64HLg1+5+0MyygO/VYzsNURh+fnwJON/M4vn49xKJzgT/95OqLHPgH42X4sfaWoHYa2aDCH6ghN0zdflDO2JmXarEn03wjxOpcjO7CfhLuI0b+bjZG4nXzOwNgm9yADcAM+oQvxr4I8EfkwHPmdnj7v5QJMHu/ncz+xRwGpBSZfl/1RK3haApPaEOuVa3nWvDpz8xszkE32xfr8MmppvZaqAYuNPMugFH67D/uWFX3RB3n2VmqUB8Hfbf0N8/NOxvuEHvvxHkmdm/gAHA9y04YFxRx210BVaa2RKCFjEA7v6Z2gLdvYgqH6TuvpP6FZqGuAH4IkHrYVfYZRnxcQV3vzVqmVWjTV0oZ2YDCa4+PIegu2MTcFP4ARZJ/FjgIWAUwTeZbsDn3T2ib8Fm1h94EJhI8A++APi2u2+OMP6XwGKC/lcjOGB7trvfG2H8MmCCux8J59OAhe4+JsL4RwiOOVwEPAF8Hlji7jUehzGz+e5+rpkVEn6wVb4EuLtnRLL/xmBmnYBD7l4efsBnuPuuCGO/CtxB0LUyyMyGAI+4+8URxvenAb//cBvV/Q1/qQ5/Q/V+/w1lZnHAGQTfmJMJPux7RfoFJdzGBdUtd/e5jZJkM2dmKcBtfPJL2leisr82ViCSCT7U+hM01Q4RfEDV+A34pG0kAMMIPtzWuHtpFFI91b7fc/exJy1bVocP+OXAOHc/Gs6nEPTrj44wfpm7j6ny2B74h7vX6UBjLJnZKGAkJ/5zRXSQ0MyWEvT/L3b3M8NlyyP9+TWmsLjHuXthBOtOcvc3zeyz1b3u7lHpnqgmj9sJTjToTXAc6WyCLyiTagxsBRrrS5KZ/Y2gJ+CLBMe0bgJWuXtDz1CsVlvrYpoGHATeo25911WNJygwCcBYM6vLB0w34KtV4oHaq7+ZfZ3g4ObAsBVQKZ3gW2ikniY48+mlcP4a4Mk6xBeHj0Vm1pOgL3RAHeJjysx+DFxIUCBmEJyiOZ8IzyIBStz9WGW3efhlIeJvWA359mdmd59ieeU2flND+AUEB9g/XbnLynCi2H9djcqz0Ba5+0VmNpwIj+s0p1Zofbj7ueFjegM3NdjdrzOzq919qpn9meB076hoawWit7tfXt/g8BS7QQTffir7jp3IP2CmAW8Ds6hb3/OfCc6g+jlwX5Xlhe6+P9KNuPtvzOwtPu6iutXd369DHq+EB4l/RVBknaCrqaX4PMGZM++7+61mlknd8p9rZv+P4HTfSwiK9vQ6xD9H8O3vMqp8+4swtqYPlhqLlLv/OHz6deBznPgFpSm7EI66+1Ezw8yS3X21mQ2LJLARP2Bbusoei4Nha3gXwe8zKtpaF9NjwEPuvrye8auAkV7PH5qZLXX3M+oT29yE3XUp4RkuLYKZvevu48wsj+A4SiHBKaKnRRgfR9ACuJSgwL7h7o/XYf/vu/uZVbroEsNtRNzFYmYT3X1BbctOEfs6H7egj3/BqaX10WjCluutwLcJzsI5ACS6+5VNsf/WIOym+zswhqBHoD3wn+7+SDT21yZaEGHfuxO831vNbCPBGRCVzdOI+vBp2Cl2EHwDv9Ld63LmUbNiZudQ5RtoXbrYmoF3wxbQ40AecJjgwsNIfdPdHwzjgeDir3BZJBrj299DwMnDQ1S3rDoNakE3VCOchdbmuXtli3cuMDDa+2sTLQir5VL22s5iMrPpBAUmneAsjDqfYhdupxBIC2NLaSH9p5VO1cXm7v8eu6wiF+Y/j6Cb7yjBGTx1uRK4upME3q88YB1BfOW3v9HAMwTf/n7k7o9GEDuB4MylbwMPVHkpA7jW3U+PYBsNakFL7IXdoj8Derr7FWY2kuDMxLocS4xYm2hBRHoaaw1+TfBh/kuCA7uVKpdFmke6BUNbDKHKQcoWJIcGdLE1A08THH95iODb11Izm1dbC8DMbiQ4a2SAmb1c5aUMggP1kXqOj48BTA2XZUYYm0RQUBI48XjEIYJjK6fUiC1oib1nCP6OfxDOrwX+St1ONolYmygQDVV5jrWZJZ58vrWZtYt0O6c4ze8dIKLz6JuBhnaxxVR4qudcgjNpLgL+jeCMotq6iBrrSvBpBBdW5lGlBRqJ8O9urpkVu/sJV16b2XXAuhrCr6rLvqRZa+hoDnWiAhGBRjzNtN6n+cXSSV1s9bqKtTkws9kEXXwLCbqZxvmJQ5dUK2yBbjGzyUCxu1eY2VBgOFCX7prGOAbwBT45NMf3gb+dKqARWtDSfDR0NIc6UYGITKOcZkoDTvOLsUbpYmsGlgHZBFfCFxAcLF7o7sU1hx03DzgvvBp5NpBLMHTCTRHGv2Nmo+tzDMDMriAYw6iXmf2uyksZ1G08MWnZ7iYYrG+gmS0gHM0hWjtTgYiAN95gZQ0dbC4mGquLLdbc/TsA4RXgtxL05fYgGPYhEubuRRYM8f6Qu99vZrVeR9JIxwB2EBSkzxB0UVUqBL4TYf7S8q0EXgKKCH73/yQ4DhEVbeIspubIgjFlOgCvu/uxWOdTk6pdbARDJVdKBxa4e0T3VIg1M/sGcB5BK2IL4RlN7v5mhPHvE/wcHiAYbO1Di2CojYaeRXfSthLcXS2GNsrMXiA4MeFP4aIbgU7ufl1U9qcCIbUxsw5AJxrexRZTZvY9gqKQV58P2bCo/wdBUfylBQPnfbspTvM1sxfc/foqrZET6EyktsHMPjj5lObqljXa/lQgRJo/M8ty952nao3oQHTbYGbPEIwgvCicPwuY4u53RmV/KhAikQmv/q3u23urH41UYqtKyzGRYDTpreF8P2Clu4+Kxn51kFokct+t8jyF4KK3JjkeUM0opsdfogVdjS/1FpNrWdSCEGkAM5vr7tXexEakpVMLQiRC4TApleIIhh7pEaN0RKJOBUIkcnl83M1TBmwmGP5bpFVSgRCJ3EiC6yDOJSgUbxNcvCbSKukYhEiEmvoiJZFYU4EQiVBTX6QkEmtxsU5ApAV5Pxw9Ezh+kVJdRvMVaVHUghCpRawuUhKJNRUIkVo05mB7Ii2JCoSIiFRLxyBERKRaKhAiIlItFQiRkJn9wMw+NLNlZrY0PEspWvt6y8xyorV9kcagK6lFADObQDBi5lh3LzGzrkBSjNMSiSm1IEQCWcBedy8BcPe97r7DzP7TzN41sxVm9piZGRxvATxgZvPMbJWZjTOzf5jZOjP773Cd/ma22symhq2SF80s9eQdm9mlZrbQzN4zs7+F98zGzH5hZivD2F834c9CBFCBEKn0L6CPma01sz+EtxcF+L27jwuvdWjHiePyH3P384FHgGnAXcAo4BYz6xKuMwx4LLwl6CGCsZyOC1sqPwQmu/tYgrGd7g5Hjr0WOC2M/e8ovGeRGqlAiADufhjIBu4A9gB/NbNbgIvMbHF4sdwk4LQqYS+Hj8uBD919Z9gC2Qj0CV/b5u6VV1v/kWCgv6rOJhgEcIGZLQWmEFyAdwg4CjxhZp8FihrtzYpESMcgRELuXg68BbwVFoSvAWOAHHffZmY/IbiTXKWS8LGiyvPK+cr/rZMvNDp53oCZ7n7jyfmY2XjgYuALwDcICpRIk1ELQgQws2FmNqTKojOANeHzveFxgc/XY9N9wwPgEIz+Ov+k1xcBE81scJhHqpkNDffXwd1nAN8O8xFpUmpBiATaAw+ZWUeCmwGtJ+huOkjQhbQZeLce210FTDGzR4F1wMNVX3T3PWFX1vNmlhwu/iFQCEwzsxSCVsZ36rFvkQbRUBsiUWJm/YFXNJiftFTqYhIRkWqpBSEiItVSC0JERKqlAiEiItVSgRARkWqpQIiISLVUIEREpFoqECIiUq3/DyI+mAko7lwxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline \n",
    "\n",
    "token_freq.plot(20, cumulative=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see from the plot, the words like he, the, and, etc. are highly populated in the text but don’t add any extra information in a sentence. Such words can often create noise while modelling. Such words are known as ***Stop Words***. \n",
    "\n",
    "Let's see the stopwords in English language as in NLTK package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'hers', \"isn't\", 'me', 'its', 'up', 'most', 'd', 'himself', 'while', 'doing', 'nor', 'under', \"needn't\", 'when', 'as', 'because', 'yours', \"you'd\", 've', 'their', 'are', 'between', 'with', 'needn', 'same', 'weren', 'this', 'very', 't', 'being', 'm', 'before', 'those', 'mustn', 'couldn', 'to', 'her', \"you're\", 'him', \"haven't\", \"couldn't\", 'should', 'just', 'his', 'from', 'ourselves', 'it', 'after', 'you', 'now', 'if', \"you've\", 'for', 'few', \"should've\", 'has', \"hadn't\", 'wasn', 'during', 'she', 'than', 'which', 'itself', 'our', 'into', 'further', 'too', 'aren', 'other', 'yourself', 'my', 'more', 'who', 'against', \"you'll\", 'or', 'was', 'and', 'hasn', 'hadn', 'off', 'any', 'theirs', 'some', 'no', 'mightn', 'whom', 'been', 'had', 'through', 'myself', 'these', 'down', \"aren't\", 'shouldn', 'at', 'ours', 'a', 'isn', 'do', \"hasn't\", 'having', 'each', 'your', 'below', \"won't\", 'both', 'what', 'how', 'an', 'so', 'shan', \"wouldn't\", 'did', 'ain', 'the', 'again', 'but', 'can', \"she's\", \"it's\", 'is', 'here', \"doesn't\", 'o', \"shouldn't\", 'haven', 'own', 'once', 'll', 'over', 'then', 'wouldn', 'in', 're', 'we', 'themselves', 'am', 'there', 'didn', 's', 'ma', \"didn't\", 'doesn', 'will', 'herself', 'about', 'i', \"wasn't\", 'they', 'until', 'not', \"weren't\", 'that', 'such', 'out', 'why', 'only', 'by', \"that'll\", 'them', \"don't\", \"mustn't\", 'above', 'won', 'where', 'on', \"shan't\", 'be', 'y', 'does', 'all', 'yourselves', 'don', 'have', \"mightn't\", 'were', 'of', 'he'}\n"
     ]
    }
   ],
   "source": [
    "stop_words = set(nltk.corpus.stopwords.words('english'))\n",
    "print(stop_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['adventures', 'buster', 'bear', 'thornton', 'w', 'burgess', '1920', 'buster', 'bear', 'goes', 'fishing', 'buster', 'bear', 'yawned', 'lay', 'comfortable', 'bed', 'leaves', 'watched', 'first', 'early', 'morning', 'sunbeams', 'creeping', 'green', 'forest', 'chase', 'black', 'shadows', 'yawned', 'slowly', 'got', 'feet', 'shook', 'walked', 'big', 'pine', 'tree', 'stood', 'hind', 'legs', 'reached', 'high', 'trunk', 'tree', 'could', 'scratched', 'bark', 'great', 'claws', 'yawned', 'seemed', 'jaws', 'would', 'crack', 'sat', 'think', 'wanted', 'breakfast', 'sat', 'trying', 'make', 'mind', 'would', 'taste', 'best', 'listening', 'sounds', 'told', 'waking', 'little', 'people', 'live', 'green', 'forest', 'heard', 'sammy', 'jay', 'way', 'distance', 'screaming', 'thief', 'thief', 'grinned', 'wonder', 'thought', 'buster', 'one', 'stolen', 'sammy', 'breakfast', 'stolen', 'breakfast', 'one', 'else', 'probably', 'thief', 'heard', 'chatterer', 'red']\n"
     ]
    }
   ],
   "source": [
    "# filter each sentence by removing the stop words\n",
    "def remove_stopwords(tokens):\n",
    "    return [word for word in tokens if word not in stop_words]\n",
    "        \n",
    "print(remove_stopwords(tokens)[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's find the frequencies and plot again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEiCAYAAADjxEWuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VOXVwPHfyU5IWMMSFYggoIiAJFpEwL3Fpe61WrVYtdRq69Za69u+rb611VartYv73tYNd3AXWQWUsEUQZFNk39cEyHbeP547MMRJcmcmk5tkzvfzuZ9k7sy595lk5p57n+2KqmKMMcbUlBJ0AYwxxjRNliCMMcZEZAnCGGNMRJYgjDHGRGQJwhhjTESWIIwxxkRkCcIYY0xEliCMMcZEZAnCGGNMRJYgjDHGRJQWdAHikZeXpwUFBTHH7969m1atWlm8xVu8xSdV/KxZszapaqd6X6iqzXYpLCzUeBQXF1u8xVu8xSddPFCsPo6xVsVkjDEmIksQxhhjIrIEYYwxJiJLEMYYYyKyBGGMMSYiSxDGGGMiSsoEsWHnHq5/fg7rSyuDLooxxjRZSZkg7nt/MW/OW8O/Zu6gutruyW2MMZEkLEGISDcRmSAiC0VkgYjc4K2/R0QWiUiJiLwmIu289QUisltE5nrLw4kq269GHk5eTgYLNpbz7xkrErUbY4xp1hJ5BVEJ/EJVjwCGANeJSD/gA6C/qg4AFgO3hcUsU9VB3nJNogrWoXUGd557FAB3v7OIFZtLE7UrY4xpthKWIFR1rarO9n7fCSwEDlbV91U1VPk/AzgkUWWoy8j+XRnWLYvdFVXcMqbEqpqMMaaGRmmDEJEC4GjgkxpPXQm8E/b4UBGZIyKTRGR4ost19dFtyMvJ5NOvtvD0tK8SvTtjjGlWxM3blMAdiOQAk4A/quqrYet/AxQB56uqikgmkKOqm0WkEHgdOFJVd9TY3mhgNEB+fn7h2LFjYy5bWVkZ87em8Odp28hIhb+elsdBuf4nuC0rKyM7Ozuu/Vu8xVu8xTd2fFFR0SxVLar3hX5m9It1AdKB94Cba6wfBUwHsuuInQgU1bX9hprN9cYX5miPW8fpBQ9+rJVV1VHHx7t/i7d4i7f4xown6NlcRUSAJ4CFqnpf2PqRwK3A2apaFra+k4iker/3BHoDyxNVvnC//24/OudmUrxiK099/GVj7NIYY5q8RLZBHA9cDpwc1nX1DOCfQC7wQY3urCOAEhGZB7wMXKOqWxJYvn3aZWdw1/muV9M9733Bso27GmO3xhjTpCXsjnKqOhWQCE+9XcvrXwFeSVR56nPKEV24YPAhvDJ7FbeMmceYa4aSmhKp+MYYkxySciR1bX733X50aZPJ7K+38cTURqndMsaYJssSRJi2rdK5+4IBANz7/mKWbtgZcImMMSY4liBqOKlvZy4qOoTyymp+MaaEyqrqoItkjDGBsAQRwW/P6kd+2yzmrdzGY1OsV5MxJjlZgoigTdb+qqb7P1jM4vVW1WSMST6WIGpxQp9OXHJsN8qrqvnlmHlW1WSMSTqWIOrwP2ccwcHtWlGyajuPTLZeTcaY5GIJog65Wen82atq+tuHi1m0bkc9EcYY03JYgqjHsN55XPqt7lRUKb94aR4VVtVkjEkSliB8uM2ralqwZgcPTVwWdHGMMaZRWILwISczjXsudFVNfx+/hM/XWFWTMablswTh09DD8vjhcT2orFZ+OWYe5ZVW1WSMadksQUTh1pGH061DKz5fu4N/TVgadHGMMSahLEFEoXVmGvdcOBCAf01YyvKtFQGXyBhjEscSRJSG9OzIKK+q6a0lpUEXxxhjEsYSRAwu+VZ3AErWl4duj2qMMS2OJYgY9O2SS15OJlv2VLN0g919zhjTMlmCiIGIMOywjgBMXbop4NIYY0xiJCxBiEg3EZkgIgtFZIGI3OCt7yAiH4jIEu9ne2+9iMjfRWSpiJSIyOBEla0hDOvdCYCpSyxBGGNapkReQVQCv1DVI4AhwHUi0g/4NTBeVXsD473HAKcDvb1lNPBQAssWt2GH5QEwY/lmm37DGNMiJSxBqOpaVZ3t/b4TWAgcDJwDPOO97BngXO/3c4Bn1ZkBtBOR/ESVL15d22ZxSG4qpeVVzF25LejiGGNMg2uUNggRKQCOBj4BuqjqWnBJBOjsvexgYGVY2CpvXZM1oEsmAFOsmskY0wJJortpikgOMAn4o6q+KiLbVLVd2PNbVbW9iLwF3KWqU73144FfqeqsGtsbjauCIj8/v3Ds2LExl62srIzs7OyY46d+uZ37i3fTt2M6fzq5Y6Pv3+It3uItPhZFRUWzVLWo3heqasIWIB14D7g5bN0XQL73ez7whff7I8AlkV5X21JYWKjxKC4ujit+yvRPtddtb2nP297S7bvLG33/Fm/xFm/xsQCK1ccxPJG9mAR4AlioqveFPfUmMMr7fRTwRtj6H3q9mYYA29WrimqqWqWncHT3dlRVKzOWbQ66OMYY06AS2QZxPHA5cLKIzPWWM4C7gdNEZAlwmvcY4G1gObAUeAy4NoFlazDHe72ZPrbxEMaYFiYtURtW15YgtTx9SoTXK3BdosqTKMN75/G3D5cwxRKEMaaFsZHUcRpwSDtyMtNYvrGUNdt2B10cY4xpMJYg4pSemsKQnjbthjGm5bEE0QCG93btEDbthjGmJbEE0QDCG6qrq236b2NMy2AJogH06tSa/LZZbC4tZ9G6nUEXxxhjGoQliAbgpv/2qpmWbgy4NMYY0zAsQTSQYV47hM3LZIxpKSxBNJBQO8TMr7awp6Iq4NIYY0z8LEE0kLycTI7Ib8Oeimpmr9gadHGMMSZuliAaUOg2pDaq2hjTEliCaECh25DavEzGmJbAEkQDOragAxmpKXy2ejtbS8uDLo4xxsTFEkQDapWRSmGP9qjCNJv+2xjTzFmCaGCh7q42L5MxprmzBNHA9s3LZAPmjDHNnCWIBnbkQW1p2yqdlVt2s2JzadDFMcaYmFmCaGCpKcLxh9n038aY5s8SRAIMO8x1d7Xpv40xzVnCEoSIPCkiG0Rkfti6F8PuT/2ViMz11heIyO6w5x5OVLkaQ2jivmnLNlNl038bY5qphN2TGnga+CfwbGiFqn4/9LuI/BXYHvb6Zao6KIHlaTTdO2bTvUM2X28pY/7q7Qzs1i7oIhljTNQSdgWhqpOBLZGeExEBLgKeT9T+g2bdXY0xzV1QbRDDgfWquiRs3aEiMkdEJonI8IDK1WBC1UxTllh3V2NM8ySqiasjF5ECYJyq9q+x/iFgqar+1XucCeSo6mYRKQReB45U1R0RtjkaGA2Qn59fOHbs2JjLV1ZWRnZ2dkLid5ZX86M3NpCaAs+c05mstG/m4kTu3+It3uItvjZFRUWzVLWo3heqasIWoACYX2NdGrAeOKSOuIlAUX3bLyws1HgUFxcnNP7sf0zRHreO0wmL1geyf4u3eIu3+EiAYvVxDA+iiulUYJGqrgqtEJFOIpLq/d4T6A0sD6BsDSp0EyGb3dUY0xwlspvr88B0oK+IrBKRq7ynLuabjdMjgBIRmQe8DFyjqhEbuJsTuw2pMaY5S1g3V1W9pJb1V0RY9wrwSqLKEpTCHu3JSk9h0bqdbNy5l065mUEXyRhjfLOR1AmUmZbKsYe6aTemLbOrCGNM82IJIsGGH2bVTMaY5skSRIKFGqqnLtkU6qFljDHNgiWIBDu8ay55ORms27GHZRtt+m9jTPNhCSLBUlIk7CrCRlUbY5oPSxCNYF+CsPEQxphmxBJEIwjdhnTG8i1UVFUHXBpjjPHHEkQjyG/bil6dWrNrbyXzVm4LujjGGOOLJYhGMsy6uxpjmhlLEI1kWG93G1Kbl8kY01xYgmgkQ3p2IDVFmLNyGzv3VARdHGOMqZcliEaSm5XOoG7tqKpWZixv9vMQGmOSgCWIRjTMpv82xjQjliAa0fDedhtSY0zzYQmiEQ3s1o6czDSWbSxl7fbdQRfHGGPqZAmiEaWnpjCkZwfATd5njDFNmSWIRjbMpt0wxjQTliAaWeg2pB8v3US1Tf9tjGnCEnlP6idFZIOIzA9bd7uIrBaRud5yRthzt4nIUhH5QkS+k6hyBa1Xpxy6tsli065yvt5eGXRxjDGmVom8gngaGBlh/f2qOshb3gYQkX7AxcCRXsyDIpKawLIFRkT2XUXMW18ecGmMMaZ2UScIEWkvIgPqe52qTgb8jgg7B3hBVfeq6pfAUuDYaMvWXJzY1027MX3VnoBLYowxtfOVIERkooi0EZEOwDzgKRG5L8Z9/kxESrwqqPbeuoOBlWGvWeWta5FOPaILuZlpLNlSwdINu4IujjHGRCR+7pMsInNU9WgRuRropqq/F5ESVa3zSkJECoBxqtrfe9wF2AQo8AcgX1WvFJF/AdNV9T/e654A3lbVVyJsczQwGiA/P79w7Nix/t9tDWVlZWRnZwcS/2DxdsZ/uZvzD2/NpUflNvr+Ld7iLT5544uKimapalG9L1TVehfgMyAfeB84xltX4iOuAJhf33PAbcBtYc+9BxxX3/YLCws1HsXFxYHFz1i2SXvcOk6H/OlDraqqbvT9W7zFW3zyxgPF6uPY77cN4g7voL1UVWeKSE9gic/YfUQkP+zheUCoh9ObwMUikikihwK9gU+j3X5zckxBBzq3TmXt9j3MWL456OIYY8w3pPl83VoNq05S1eX1tUGIyPPAiUCeiKwCfg+cKCKDcFVMXwE/8ba3QEReAj4HKoHrVLUqyvfSrKSkCCf0yGLM56W8PHsVQ70BdMYY01T4vYL4h891+6jqJaqar6rpqnqIqj6hqper6lGqOkBVz1bVtWGv/6Oq9lLVvqr6TjRvork6sUcrAN6dv47SvTYmwhjTtNR5BSEixwFDgU4icnPYU22AFjlOoTF1zUmjqEd7ilds5d3567ig8JCgi2SMMfvUdwWRAeTgEklu2LIDuDCxRUsO5w92SeHVOasCLokxxhyozisIVZ0ETBKRp1V1RSOVKamcOSCf28cuYNqyzazZtpuD2rUKukjGGAP4b4PIFJFHReR9EfkotCS0ZEmibat0TuvXBVV4bc7qoItjjDH7+E0QY4A5wG+BW8IW0wAuDFUzzV4VGgdijDGB89vNtVJVH0poSZLY8N555OVksGxjKSWrtjOwW7ugi2SMMb6vIMaKyLUiki8iHUJLQkuWRNJSUzhnkJt66pXZ1lhtjGka/CaIUbgqpWnALG8pTlShktEFXjXTm/PWUF5ZHXBpjDHGZ4JQ1UMjLD0TXbhk0u+gNhzeNZdtZRV8tGhD0MUxxhh/bRAi8sNI61X12YYtTnK7sPAQ7nxrIa/OXsXI/l2DLo4xJsn5rWI6JmwZDtwOnJ2gMiWtswcdRIrAhC82sKXU7jZnjAmW3yqmn4ctPwaOxo2yNg2oc24WI/p0oqJKGTtvTdDFMcYkuVjvSV2Gm5LbNLALwsZEGGNMkPy2QYzFTdENbpK+I4CXElWoZHZavy7kZqUxb9V2lm7YyWGdY7vbnDHGxMvvQLl7w36vBFaoqp3iJkBWeipnDcjn+U9X8srs1dw68vCgi2SMSVJ+2yAmAYtwM7m2B6wFNYFCM7y+Pmc1VdU29YYxJhi+EoSIXIS7Bej3gIuAT0TEpvtOkKIe7eneIZu12/cwfZndjtQYEwy/jdS/AY5R1VGq+kPgWOB/E1es5CYinD/YTb1hjdXGmKD4TRApqho+vHdzFLEmBucf7aqZ3rHbkRpjAuL3IP+uiLwnIleIyBXAW8DbdQWIyJMiskFE5oetu0dEFolIiYi8JiLtvPUFIrJbROZ6y8OxvqGWonvHbI4t6MDuiiremb8u6OIYY5JQnQlCRA4TkeNV9RbgEWAAMBCYDjxaz7afBkbWWPcB0F9VBwCLgdvCnlumqoO85Zoo3kOLZdVMxpgg1XcF8TdgJ4CqvqqqN6vqTbirh7/VFaiqk4EtNda9r6qh+pIZwCExlTpJnDEgn8y0FKYv38zqbbuDLo4xJslIXXcwE5H5qtq/luc+U9Wj6ty4SAEwLtI2vMF3L6rqf7zXLcBdVewAfquqU2rZ5mhgNEB+fn7h2LFj6ypCncrKysjOzm7S8ffN2MbHK/fwg/45XHBETqPv3+It3uJbXnxRUdEsVS2q94WqWusCLI3lubDXFADzI6z/DfAa+xNUJtDR+70QWAm0qW/7hYWFGo/i4uImH//RovXa49ZxetK9E7S6urrR92/xFm/xLS8eKNZ6jq+qWm8V00wR+XHNlSJyFe6mQVETkVHAWcClXkFR1b2qutn7fRawDOgTy/ZbmuGH5dEpN5PlG0uZu3Jb0MUxxiSR+qbauBF4TUQuZX9CKMLN5HpetDsTkZHArcAJqloWtr4TsEVVq0SkJ24iwOXRbr8lSktN4dxBB/HYlC95dfZqju7ePugiGWOSRJ1XEKq6XlWHAncAX3nLHap6nKrW2fdSRJ7H9XbqKyKrvKuOf+Km6/igRnfWEUCJiMwDXgauUdUtETechEJTb4wtWcPeyqqAS2OMSRa+JutT1QnAhGg2rKqXRFj9RC2vfQV4JZrtJ5Mj8tvQL78Nn6/dwYRFGxjZPz/oIhljkoCNhm4mQmMiXpm9OuCSGGOShSWIZuKcQQeTmiJMWLSBzbv2Bl0cY0wSsATRTHTKzeSEPp2orLbbkRpjGocliGZk39Qbc6yayRiTeJYgmpFTj3C3Iy1ZtZ0l63cGXRxjTAtnCaIZcbcjPQiwxmpjTOJZgmhmLvCqmV6fs5qqOubRMsaYeFmCaGYKe7SnR8ds1u3Yw2fr7dbgxpjEsQTRzIjIvrvN3T9jG2/MXR2aANEYYxqUJYhm6Orhh3Ji307sqlBueGEu1/53to2NMMY0OEsQzVDrzDSeuuIYflrUhtYZqbwzfx3fvn8y785fG3TRjDEtiCWIZkpEOPXQbN69cQTH9ezI5tJyrvnPbG58YQ7byyqCLp4xpgWwBNHMdeuQzX+v/hZ3nH0kWekpvD53Dd/+2yQmfLEh6KIZY5o5SxAtQEqKMGpoAe/cMILCHu1Zv2MvP3pqJr9+pYSde+xqwhgTG0sQLcihea156SfHcdvph5ORmsILM1cy8m9TmLZ0U9BFM8Y0Q5YgWpjUFOEnJ/Ri3PXDOOrgtqzetpsfPP4Jt7+5gLLyyqCLZ4xpRixBtFB9uuTy6rVDufm0PqSlCE9P+4ozHpjCrBV2oz5jjD+WIFqw9NQUrj+lN69fdzx9u+Ty1eYyLnx4One9vZA9FXbrUmNM3XzdcjRWIvIkcBawQVX7e+s6AC8CBbh7XF+kqltFRIAHgDOAMuAKVZ2dyPIli/4Ht+XNnx/PAx8u4eFJy3hk8nI+WrSB3m2qGbt6Qczbrdi5i3Xpa+nRMZvuHbNpk5XegKU2xgQtoQkCeBr4J/Bs2LpfA+NV9W4R+bX3+FbgdKC3t3wLeMj7aRpAZloqvxp5OKf268IvX5rHkg27WLIBWPpVXNv972f7c3i77HR6dMime8fW3s9sunfIpkfHbLrkZpGSIvG9CWNMo0poglDVySJSUGP1OcCJ3u/PABNxCeIc4Fl1EwvNEJF2IpKvqjY8uAEN7t6et28YzriStXy+ZDndunWLaTtV1crcxSvYk5bL11tK+XpLGdvKKthWtp15q7Z/4/WZaSl06+ASRvcO2RR0zKbt3goK431DxpiESfQVRCRdQgd9VV0rIp299QcDK8Net8pbZwmigWWlp3Jh4SHMYj2FhYfGvJ1Z2VspLHSHeFVl4869rNhSxorNZXy9pYyvN5eyYksZX28uY3NpOUs37GLphl0HbOPBuZM4a8BBnDUwn16dcuJ6X8aYhiWJngnUu4IYF9YGsU1V24U9v1VV24vIW8BdqjrVWz8e+JWqzqqxvdHAaID8/PzCsWPHxly2srIysrOzLb4R4ndXVLO+tIp1u6pYV1rFyu0VFK/Zy66K/Z+/grZpDO2WxfHdsuiaU/+5S3N6/xZv8U0pvqioaJaqFtX7QlVN6IJrjJ4f9vgLIN/7PR/4wvv9EeCSSK+rbSksLNR4FBcXW3yA8TM+nakfLVqvN784V/v//l3tceu4fctZf5+iD01cql9vLk3Y/i3e4pM1HihWH8fvIKqY3gRGAXd7P98IW/8zEXkB1zi9Xa39oUVLSxFO6tuZk/p2Zm9lf6Ys3sS4kjV88Pl6Plu9nc9Wb+fudxYxqFs7zhqQz5kD8slv2yroYhuTNBLdzfV5XIN0noisAn6PSwwvichVwNfA97yXv43r4roU1831R4ksm2laMtNSObVfF07t14U9FVVM/GIj40rWMH7hBuau3Mbcldu4862FFPVoz1kD8jnjqPygi2xMi5foXkyX1PLUKRFeq8B1iSyPaR6y0lMZ2b8rI/t3pay8ko8WbeCtkrV8tGgDxSu2UrxiK3eM+5zjDs7iTz1KKchrHXSRjWmRgqhiMsa37Iw018tpwEHs2lvJ+IXrGTtvLZMXb2Taqj2cet8kLv1Wd35+Sm/ycjKDLq4xLYpNtWGajZzMNM4ZdDCPjypi4i0ncnJBK6pUeWb6Ck74ywQe+HAJpXttQkJjGoolCNMsHdSuFdcd05Z3bxjByYd3prS8ivs/XMwJ90zkPzNWUFFVHXQRjWn2LEGYZq1v11yevOIYXhg9hIHd2rFp115++/p8vuPdo1sTPM7HmJbMEoRpEYb07Mjr1w7lXz8YTEHHbJZvKuWa/8zm/Iem8emXNsW5MbGwBGFaDBHhzAH5fHDzCfzhnCPJy8lgztfbuOiR6Vz9zEyWrN8ZdBGNaVYsQZgWJz01hcuPK2DiLSdxwym9yc5I5cOFG/jO3yZz68slrNu+J+giGtMsWIIwLVZOZho3ndaHibecyGVDuiMivFi8khPvncBf3l1EWYU1ZBtTF0sQpsXrnJvFnecexQc3jeCMo7qyp6KaBycu47bxm1m1tSzo4hnTZFmCMEmjZ6ccHry0kNeuHUqfLjms2lnFeQ9OY/7qb96/whhjCcIkoaO7t2fMNUPp3ymDjTv38v1HpjNp8cagi2VMk2MJwiSltq3S+e2I9pw76CBKy6u48umZvDRzZf2BxiQRSxAmaaWnCPddNIhrT+xFVbXyq1dKuP+DxTa4zhiPJQiT1FJShF+NPJw/nNufFIEHxi/hVy+X2FQdxmAJwhgALh/Sg0cuLyIrPYUxs1Zx1TPF7LKJ/0ySswRhjOe0fl14/sdD6Ng6g8mLN/L9R6azYYcNqjPJyxKEMWGO7t6eV346lIKO2SxYs4PzHpzG0g02RYdJTpYgjKmhIK81r/x0KEd3b8fqbbs5/0Gb8M8kp0ZPECLSV0Tmhi07RORGEbldRFaHrT+jsctmTEjHnEyeu3oIp/Xrwo49lVz2+CeMK1kTdLGMaVSNniBU9QtVHaSqg4BCoAx4zXv6/tBzqvp2Y5fNmHCtMlJ5+LJCfnhcD8qrqvnZc3N4fMpy6wZrkkbQVUynAMtUdUXA5TAmotQU4Y6zj+S20w8H4M63FvJ/4z6nqtqShGn5gk4QFwPPhz3+mYiUiMiTItI+qEIZE05E+MkJvXjg4kFkpKbw1Mdfcd1/Z7O3ypKEadkkqMtlEckA1gBHqup6EekCbAIU+AOQr6pXRogbDYwGyM/PLxw7dmzMZSgrKyM7O9viLd63+Rv28udp2yirULpkp3DZgDYcd0gmItIo+7d4i2+I+KKiolmqWlTvC1U1kAU4B3i/lucKgPn1baOwsFDjUVxcbPEWH7Uv1u3Qk+6doD1uHac9bh2nZ/9zqk5ftqnR9m/xFh9vPFCsPo7TQVYxXUJY9ZKI5Ic9dx4wv9FLZIwPfbrk8t6NIxg9uA15OZnMW7mNix+dwZVPz2TRuh1BF8+YBhNIghCRbOA04NWw1X8Rkc9EpAQ4CbgpiLIZ40d6agrf6ZXNpFtO5ObT+tA6I5WPFm3g9Aem8Msx81izbXfQRTQmbmlB7FRVy4CONdZdHkRZjIlH68w0rj+lNz/4Vnf+MX4J//3ka16etYqx89ZwxfEFXHviYbRtlR50MY2JSdC9mIxpEfJyMrnjnP58ePMJnDkgn72V1TwyaTkj/jKBxyYvZ09FVdBFNCZqliCMaUAFea351w8G88Z1xzOkZwe2767gj28v5JS/TuLV2auotvETphmxBGFMAgzs1o7nfzyEp644hr5dclm9bTc3vzSPM/8xlUmLN9pobNMsBNIGYUwyEBFOOrwzI/p04rU5q7nv/S9YuHYHo578lKG9OnJBTzfXjDFNlSUIYxIsNUW4sPAQzhqQzzPTvuJfE5YybdlmPl0Oa2UJ15zQi7RUu5g3TY99Ko1pJFnpqfzkhF5M/tVJXDakO5UK976/mAsesntOmKbJEoQxjaxddgZ3nnsUvxvRnoPaZjFv1XbO+PtUHpu83CYBNE2KJQhjAjKwSybv3jSCi4oOobyymj++vZCLH53Ois2lQRfNGMAShDGBapOVzl8uHMgTo4rolJvJzK+2MvJvU/j39K+sS6wJnCUIY5qAU47owvs3juDsgQexu6KK/31jAZc/+QmrbcoOEyBLEMY0Ee1bZ/D3S47mwUsH06F1Bh8v3czI+yfz0syVNm7CBMIShDFNzBlH5fPejSP4dr8u7Nxbya9eKeGqZ4rZsGNP0EUzScYShDFNUKfcTB65vJD7vz+Q3Kw0Plq0gdPun8wbc1fb1YRpNJYgjGmiRITzjj6ED246gRP6dGL77gpueGEu1z03m8279gZdPJMELEEY08R1bZvF0z86hrvOP4rWGam8/dk6vn3/ZN74opRF63bYFYVJGJtqw5hmQES45NjuDDssj1tenseM5Vt4tqScZ0um0Dk3k2GH5TG8Tx7HH5ZH59ysoItrWghLEMY0I906ZPPc1UN4Z/46xnz8OZ9vUTbs3Murc1bz6pzVABzeNZfhvfMY3rsTxx7agaz01IBLbZorSxDGNDMpKcKZA/LpWrGGwYMHs3j9LqYs2ciUJZv45MvNLFq3k0XrdvLYlC/JSEvh2IIODOudx7DD8uiX34aUFAn6LZhmIrAEISJfATuBKqBSVYtEpAPwIlAAfAVcpKpbgyqjMU08mQdOAAAbOElEQVSdiNC3ay59u+Zy9fCe7K2sYtaKrUxZsompSzYxf812pi7dxNSlmwDo2DqD4w/LY3jvPCq3lNNu4y7atUqnTat00m1GWVND0FcQJ6nqprDHvwbGq+rdIvJr7/GtwRTNmOYnMy2Vob3yGNorj1tHwpbScj5euokpSzYydckm1mzfw5vz1vDmvDUu4KNJ+2JbZ6TSLjuDNq3SadsqjXatMmjbKp222enup7e08x6v21XJ9rIKcrPS7KqkhQo6QdR0DnCi9/szwEQsQRgTsw6tM/juwIP47sCDUFWWbyplyuKNTFu2meVrt1CZks723RVs311BaXkVpeW7o5ve4533EYHczDTaZWfsTyRhSaVdWHIJX791TxUbdsY++K+sohpVRcSSU6IEmSAUeF9EFHhEVR8FuqjqWgBVXSsinQMsnzEtiojQq1MOvTrlcMXxhzJr1iwKC9097aqrlV3l7ooglDC2765gm/d42+5ydtRYt3FbKburhJ17K9mxxy1RGzs+rveU+uY7B1zdhF/hfHN9xgHPm/pJUH2oReQgVV3jJYEPgJ8Db6pqu7DXbFXV9jXiRgOjAfLz8wvHjh0bcxnKysrIzs62eIu3+Djiq6qVsgplZ3k1pRXV7CpXdpVXU1qu7KqoprQ8bF1FNTvLldLyaiqrFSG2s38F9lRWs7cq5uKTngI5GSm0zkghN0NonZ5CTobsW5eTIeSkhz2fkUJOuvuZliJN5u8fi6KiolmqWlTf6wK7glDVNd7PDSLyGnAssF5E8r2rh3xgQ4S4R4FHAYqKijR0BhSL8DMoi7d4i29+8UcNPDrsiqf8m1c+ZRX7r3x2H7iuvKqarXvcEq3WGamkUk1aeuxVZCnVleS11QOueEJXOW1qVs15Vz65Wemkeu098f79/AgkQYhIayBFVXd6v38b+D/gTWAUcLf3840gymeMaR4y0lLolJtJp9zMqOJUlemfzuLQw490SaVsfwIJr2bbv678gKq30nLv0qW8PK7yb9od/a1mc7PSaJedTk5KJe8kNj8EdgXRBXjNa1xKA55T1XdFZCbwkohcBXwNfC+g8hljWjARITNNyG/bivy2raKKDbXXFM+ey8ABA2LavwIzZ8+lW6/DI7b51LwiCj23c0/lvqVjq8R3Sw4kQajqcmBghPWbgVMav0TGGONPSorQJiudtpkpdMyJ7solXOfWafQ/uG1UMVXVys49LlnMKfks5n371dS6uRpjjKlFaorQLjuDdtkZbG6b+J5YNnTSGGNMRJYgjDHGRGQJwhhjTESWIIwxxkRkCcIYY0xEliCMMcZEZAnCGGNMRIFN1tcQRGQjsCKOTeQBm+p9lcVbvMVbfMuK76Gqnep9laom7QIUW7zFW7zFJ2O8n8WqmIwxxkRkCcIYY0xEyZ4gHrV4i7d4i0/S+Ho160ZqY4wxiZPsVxDGGGNqYQnCGGNMRJYgmhERGSYiP/J+7yQihwZdJmNMy5U0bRAikgKUqGr/gPafCtytqrfEGP97oAjoq6p9ROQgYIyqHt+Q5UwU7/0/o6qXxbmdV4AngXdUNeq7zYvIwUAPwm6WpaqT4ylTFPv+t6peXt+6CHGD63peVWdHUYZUVa3y+/qwuLi/PyLyPVUdU9+6pkxEpgCTgSnAx6oa/U2lY9tvg30GopE0d5RT1WoRmSci3VX161i3IyJ9gIeALqraX0QGAGer6p317L9KRApFRDS2rHwecDQw29veGhHJbYyy19jG+cCfgc6AeIuqapu64rz330lEMlQ1nju9PwT8CPi7iIwBnlbVRT7L/mfg+8DnQOggqbgvvJ/4eP9+R9bYXirg57bzf/V+ZuFOEubh/u4DgE+AYT73D7BURF4GnlLVz/0GNdD35zagZjKItO4AIjIW93+qrWxn+y2AiPw9wurtuEFnb/jYxCjc3/sC4B4R2QtMUdWbfOw7ns/PX+t4ToGTfWwjakmTIDz5wAIR+RQoDa2M5gMGPAbcAjzixZaIyHOAn3/yHOAN78AWvv9XfcSWq6qKiAKISOsoyhwST9lD/gJ8V1UXxrD/r4CPReRNDnz/9/ndgKp+CHwoIm2BS4APRGQl7r39R1Ur6gg/F3cFtjeGskOMfz8RuQ34H6CViOwIrQbK8dFVUVVP8rbzAjBaVT/zHvcHfhnlexgAXAw87l0VPAm8oKo76g4DYvz+iMjpwBnAwTUO0G2ASh/7vdf7eT7QFfiP9/gS3GcqGlnA4exPShcAC4CrROQkVb2xrmBVXS4iu3H/u3LgJOAIn/uO+fsX+gw0tmRLEHc0wDayVfVTEQlf5+dDDtAB2MyB2V4BPwniJRF5BGgnIj8GrsR94KIRT9lD1seYHADWeEsKENXVTzgR6QhcBlyOS7r/xZ3VjQJOrCN0OZAOxJogYvr7qepdwF0icpeq3hbjvgEODyUHb7vzRWRQNBvwqkQeAx4TkRHA88D93lXFH1R1aR3hsX5/1gDFwNnArLD1O4F6z7xVdRKAiPxBVUeEPTVWRKKtHjwMOFlVK71tPgS8D5wGfFZXoPf6Zbj5j54DngB+HkVVZ9zfPxFJB34KhP4OE4FH6jkxillSJQhVnSQiPYDeqvqhiGQDqVFuZpOI9MK75BWRC4G1Pvf/oyj3FR57r4icBuwA+gK/U9UPotxMzGUPUywiLwKvE3ag9XMVpKp3ePttraql9b0+EhF5FXcG+G/clUyo/C+KSHE94WXAXBEZX6Ps1/vcfbx/v3Gh9y4ilwGDgQdU1e+EkwtF5HHcGbTikmRUydqr1joTV01XgKu6+C8wHHgb6FNbbKzfH1WdB8wTkefiPJB1EpGeqrrcey+HAvVPOHegg4HWuGolvN8P8qpA/Zw4/B13MnIJrsp3kohMVtVlPmIb4vv3EO4k50Hv8eXeuquj3I4/iZ7sqSktwI+BmcAy73FvYHyU2+gJfIg72KwGpgIFPmOzgOu8f+6ToSWKfXcBzvKWzjG8/5jLHraNpyIsvt4DcByu/v9r7/FA4MEo939yHP//UZGWxvr7ASW4qqWB3u83AJOiiM/CnXG/5i03AVlR/g2W4858h0Z47u/1xMb1/fFe/7L3GVgeWqKIHwl8jTtrnoirXvpOlO//KuBL73P7tFeGq3GJ4p4otpMD/Bw3m3RVY3x+vG3M87OuoZaEbLSpLsBcIAOYE7busxi31RrIjTJmDPAHYJl3cHofdwbpJ/Yi78P4DPCs9yG/sLHKHhYb1QGpRuwnQLcaf//5UW4jHbjeO9C87H1J06OIb4Vrh4jncxTT3w+Y7f38HXBV+LrGWoBDIqw71GdsXN8f74B4Ci459gBuB+6IsvyZuAQ7EMiM8W+QD5yDa5M6KMrYv3qf4wW4RDsK6NkYn5/Q5wXoFfa4ZyI/Q0lVxQTsVdXyUB2giKRRR++IcCJycy3rAd8NrYep6vdE5BxVfcZroHrPV8nhN8AxqrrB228n3NnIyz7jEZFMXKNcAZAWVvb/87sNYL6IrMd185uM6+q3vZ6YfVR1ZY062Gi7XMZ8iS0i38U1eGYAh3r19/+nPjspiEgX4E+4g8rpItIPOE5Vn/BZ9p1eg/XlwHCvuifdx34/o+5ePAN87h9cVdzp6jVKe+/hJcBP99WYvz+eVqo63uvJtwK43es2+vu6gkTkZFX9yOtBF66XiKA+qjdF5HBVXRTWXXSl97OriHRV/91EZwB/UdX1Pl8fXoZ2wA/55vfPbxUnuEbuCSKyHHc12gNXXZgQyZYgJolIqDfJacC1wFifsXU1qvr9koTqX7d5PVDW4T4sfqSEkoNnM9EPdHwDV/c6ixgbalX1MBHpjquzPgt4UES2qaqfxtKVIjIUUBHJwF0JRNvgfYyqDgx7/JGIzPMZeztwLK56AlWdK9ENNnwaVzXxG+/xYuBF3JmkH98HfgBcqarrvL/jPT7izoqijPX5E65x90xcW9azwKU+Y+P5/gDs8XpOLRGRn+GqWTr7iDsB+Aj4boTn/HbyuBkYjbsCCP++ClF0E1XVMSJyttfAD66K0O/f4G1cgvkMiHoMj7f/8SLSG/e/E2CRxt4rz9cOk2bBHVB/jKvqeRn4cQzbON7Pulpirwba4z7wy4ENwDU+Y+/BXW1c4S3v4s5koil7VNU5tWzjEFwD3cPAdOAt4DafsXm4BtH13nv/D9Axyv3HfIkNfOL9DK8iKYli3zMjxM+Nsvw9gFO937OJsaohzv/hucA03IGqdxRxcX1/gGNwdfeH4BLtK8CQRn7vrYBf4NpwXiXKdhzgLmA8rhfhlcAHwF0+Y+OuCuKbVaw/I4oq1miXpBlJDSAiN6jqA/Wtq2cbs1V1cH3rEsG7xD4ed+YwWVVfjzL+UeAfGtZVMoYyVOMaKv+k/gYWNSgROQV3cFnurSoAfqSqE3zEPoH7cv8aV9V2Pe7LdY3PfU/04j5Q1cEiMgT4s6qe4DP+x7iz2A6q2ss7E3xYVU/xGb+T/We/GbiDRanWM0jRi/0HB545n4z7G34F/qo5GuL748XE04vtTNyAw6zQOo2iilREXsL1BPyvt+oSoJ2qXuQzvgQYpF7XVq+acI76qOYTkZuAXcA4DuxFtyWK8j+O+78/4626HNdInpBeTMmWICId3Oeo6tE+Yo8DhgI3AveHPdUGOE8PrPaobRtR12GLyFRVHRZ2cAivwK8GtuB6XzwYcQMHbutzXE+S5bgPaGgUtO86bBEZiOvmNwLoDizBXWbXW83iVef8HK8ONrReoxsJm4U7AwwdVD8A7lfVPT5is3HVQ9/2Vr0H3Okn1osfDPwDV18/H9fF8kJVLfEZPxdXxfVJ6DMnIp+p6lF+4iNs71zgWFX9Hx+vHVXX86r6TF3Pe9uI+fvjvfY4XHVcjqp29z5LP1HVa33GP4y76joJeBy4EPhUVa/yE+9tY17N72qkdXXElwAnhg7qItIBmOgzQVwH/BHYxv5kraras7HKH62kaIMQkUtwdb+HihvFG9IGV5fvRwbu8jiNA9sjduA+qH48TZR12Ko6zPsZsQ1E3KCxaexvtK3L6bgqruHe48m4D6tvqjpP3GChZd52LsMlCz/18K97rxtLjHWwuDrzHbjeYODOAP8NfK++QFUtA34jIn+K8Qy2F+5v2A13JfEtovsOxdvIewBVfV1Efu3ztc94+2wN7FFvPibvDDizrtg6vj+5+P/+APwN+A7wplemeWF1+X4MVdUBIlKiqneIyF/x1/4Qbo6IDFHVGQAi8i3g4yji7/K2MQF3gjUCN12IHzfjOqpsiqbANVSJSC/1xl2ISE+i7+jhW1IkCNwBdC2uDjx8TpOduC539VI3mnOSiOxW1b+EPyci38OdSdcnT1Vf8nqyoKqVIhLXP1dVN4vIiT5ffi6uHeRV3If737hRtf/wuz9xg9EycX/TqcAI9T/Qa4+qRpoLJxp9a5wtTfDbSO01kD+OS/RRn8EC/6uukbI9cCrus/QQLlH4EVcjb41ePCm4eZmiTTDjcWXf5T1uhetuPbSOmLi/PyEaXy+20JVembjJKrcAvjoZyP6eYOnAD0Xka+9xD9y4DF9U9XmvqvEY3HfoVlVd5zN8AW4MRDzCezGBV8Ua5zZrlRQJwjuArRCRU4Hd6iYe64MbkRttffzFuPmIwtU74Zin1DvjD42kHML+EZ0x0/2jietzFa5RsNTb/59xDc2+EwRwuqpujLKIIQ+Im5X2fQ6sg41mJsp4zgDvJ74z2NDB7Exc28EbInJ7FPG/xv0PPgN+guvV8ngU8eG9eCpx7QfnRBEPrkE2lBxQ1V1e1Vutwr4/k70TpX28z9CtPvcdby+2seK6it6D66yg+J9upiF7gqXgpttIA/qISB/1NyNwFW4k/wRiG8kP7rP+CPurWB/BfYcTIikSRJjJuP7n7XFnUsW4rof1dvOT+CccA3eJ+SbQU0Q+xqvD9l/8uAkHnrFVcWCbhh/lInIf++eCmYQbS+An0R2Fa1Q7mf1VTL66GDbgGWA8Z7Crxc2HdSrwZ3HjSnx1NZYDpzuPdg4tIL6pWsKUisjgUFIWkUJgt8/Y0/hmMjg9wrraXAM8gJvuYhXuROE6n7EAi3ANsq947XeDcdWW9YriKrdOsn9G4AUc+Bn2kyBex2d56xBzFWsski1BiKqWichVuN48fxGROT5j45pwzPM5rntdmRf3Oq4dorE8BXwiIq95j8/Ffx/+kCdxDbShXh+Xe9utOYgpkvNwo05jme67Ic4A4z2DvQg33cO9qrpNRPJxl/z10gaY7lxEDsFd7R2POyhNBW5Q1VVRbOZGYIyIrPEe5+MOeHXt96e46rBeXiNtSC6u+slP2VOBy1XV75iLSEJVfMNwySraKr6GEPOMwH46AvgQcxVrLJKtF9Mc3Af9ftxUBwui7UUiImnqzQQZw/4jdbFrr6oJyf61lGEwrhdSqKus3wQZip+rNQbFRVpXS+yLuNkvN9T32kQQkTzcGeypuPf/Pu4AG01Dazz7fwR31hvTdOci8gFuFtF/e6suAy5V1dOiLEc6Bw60qnMCPXFTq7fHNdDezf6rx6nRfH5EZKKqnhhNWWvEz1HVo0XkLtwUH89F04uqIYjIO8D3wqvpfMS8pKoXSeQR8RpNDyQReRpXvRlexToqina0qCTbFcSNuPaC17zk0BOot/887P8n4+rAv5FVfXYVbdTsH4lXtRDP3ad2i8gwVZ0KICLH47+KoguwSERmcmAdbDT344hJA53Bxive6c47qepTYY+fFpE6719Qi75AP9xYgqPFTVfxbG0v9qoPt4vIDNzgxlAnh2dE5DFV9duG9bGI/BPXcy88Qfr9PMZcxdeAYpkR+Abv50IOvOIUvtmeWZ9vsb+KFVxX84Wh5BNNl3U/kuoKIh4ikq+qa8VNd/wNfuo4Gzv7J4LX8+dZoK23aivuPdTbm0VEIg4oq9nwmSjxnsE2YDlycV9m32ehXtyHuK7Sz3urLsENEvQ10M7bxu9x98zoh2skPx13JVBvW5hXvXRcWCeH1sB0vwclr3EW9p9Fh8bh+JrmwmtMH4m7eljiVfEdparv+4lvCCLyS6BmJ402fpKkRB5HUhLNQb22409IQ7W17NtfMiUI7wMa6ew/IbfrC9tveANrX9yUxfsaWDWg+2RHS9w8Ohd6XXXbAKi/O5GFzuDfU9VTE1nGesrwR1xii/UMNt7998dVD3XwVm0CfqiqC3zGdwf+iZs2XXH1/9drFLcA9T6LA3GjfweKG7z5uKpGmucoUuwx6g0sFDdocabfKloR+QUHDvZUXJVrsarO9fsegiQis3EnRKG7+l0C3KiqtbaDhLXh9MSNHwrJxU12Gdd92hMp2aqYwm/PmIUb7OSrPUEOnObggKdwZ0F1TXfQkF3sAuN1D/4Z8JLfxBAWWyUiZSLS1mePp0QI9fUP3RktqonaGsCjwM3qTQsibvzKY9Q9BiHcH3AHp61efAfc7LRXRlGGUDfvSi/Jb8AduPyIt5NDIW7sxpu4v/2ZuGlbfiIiY7TG+KIm6kLgZRG5FNeW90P2j8yvzXPAO7g2nPCBjTs1imk2gpBUVxCRiMgk9TmXjgER+V9cm0PNs/B6P+heI/0Q3PQY4bHR9AOPWdBnsBL/NA/faJCNtpFWRB7E3R/7YtyUJbtwEw766kIbTycHEXkPuCBUtSYiObgJ584DZqlqP7/bCpK4MVSv46YMP1dV/bbBNTtJdQXhnXGFhEaidg2oOM3VlbgDa812Ez9noW95S1CCPoNd7iXY8F5IX0YRnyIi7WtcQUT1HQ5r73pYRN7F1Z/7Hg0dZyeH7kB4F98KoIeq7hZ/t/sMTIQeSB1wt1v9xGvkb9DG4aYiqRIEbvxC6J8cGonqe6IvA7jGzWtxZ5GKu3HQw34CG6gfeDw6AoPDzmB/jzuDHYH7bCQkQYjIv1X1ctzfqoD9vYAmEd00CX8FponIy7i//UW4yd/8lKHW2YYlbOBcgj0HzBCR0CzA3wWe9xq7fQ92DEiLqCaOVlJVMYlIK755cHtIfc7maWody+FrumRx01vfxf4ulgBoFLNZxkNEFgID1Ruo5nWTnKuqRySyP724WXRPx125nMT+tg8g6ume++HaTAR3P2hfB9awHkRw4JlwVD2J4iVu5HaoimqqqhY3xn5NbJLtCuIZ3MEtNFVGQoept1DxjOV4Cnd7yftxB8ofEf1UH/EI6gz2YdwNnnriRuOHhBKF7wTpJYSoy6qqJ0HtJ0nRbi9WqjqLA2ciME1Ysl1BNOpc6i1RPGM5RGSWqhZK2Oh1EZmiqsPri20oQZ7BishDqvrTxtpfLWWI64Y5Jrkk2xVEvHPBJy1pmMnyYr0ncYMJ8gw26OTgCXw0v2k+kiJBNNDBLdnF3EgX1kj7Bu6OYNfj+vSfDNR5pzPT4OwkyfiWFFVMjT083RyoRiPtidRod2jqg4VagpYymt80rqRIECZYInI98FNcY+xq9jfOhnrQNEovpmRmJ0kmFpYgTKNpCo20xhj/LEEYY4yJqLHnUjfGGNNMWIIwxhgTkSUIYzwi8hsRWSAiJSIy1+sCmqh9TRSRokRt35iGkBTjIIypj4gchxvrMVhV94q7f3VGwMUyJlB2BWGMkw9sUtW9AKq6SVXXiMjvRGSmiMwXkUdFRGDfFcD9IjJZRBaKyDEi8qqILBGRO73XFIjIIhF5xrsqeVncbTMPICLfFpHpIjJbRMZ490lARO4Wkc+92Hsb8W9hDGAJwpiQ94FuIrJYRB6U/ffP/qeqHuMNJGvFgSPKy1V1BG4yvjeA64D+wBUi0tF7TV/gUe9+ATuocR8N70rlt8Cp3v2Ki4GbvXs9nAcc6cXemYD3bEydLEEYA3j3iCgERuNuSv+iiFwBnCQin3gjkU8GjgwLe9P7+RmwQFXXelcgy4Fu3nMrVTU0lcV/cBMFhhuCm/78YxGZi5t6pAcumewBHheR84GyBnuzxvhkbRDGeFS1CpgITPQSwk+AAUCRqq4UkdsJu48FELoLWnXY76HHoe9WzYFGNR8L8IGqXlKzPCJyLHAK7vagP6Px7p1tDGBXEMYAICJ9vRsahQwCvvB+3+S1C1wYw6a7ew3g4KbWnlrj+RnA8SJymFeObBHp4+2vraq+DdzolceYRmVXEMY4OcA/RKQd7na0S3HVTdtwVUhf4e5fHa2FwCgReQRYQo2b86jqRq8q63nvDnfg2iR2Am+ISBbuKuOmGPZtTFxsqg1jEkRECoBxNlOqaa6siskYY0xEdgVhjDEmIruCMMYYE5ElCGOMMRFZgjDGGBORJQhjjDERWYIwxhgTkSUIY4wxEf0/V62wX38gcosAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "token_freq = nltk.FreqDist(remove_stopwords(tokens))\n",
    "token_freq.plot(20, cumulative=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, now the plot makes much moree sense and we can easily understand that the text is talking about ***bear and farmers***.\n",
    "\n",
    "Now let's tokenize this txt by sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[the adventures of buster bear by thornton w. burgess 1920]\\r\\n\\r\\ni\\r\\n\\r\\nbuster bear goes fishing\\r\\n\\r\\n\\r\\nbuster bear yawned as he lay on his comfortable bed of leaves and\\r\\nwatched the first early morning sunbeams creeping through the green\\r\\nforest to chase out the black shadows.', 'once more he yawned, and slowly\\r\\ngot to his feet and shook himself.', 'then he walked over to a big\\r\\npine-tree, stood up on his hind legs, reached as high up on the trunk of\\r\\nthe tree as he could, and scratched the bark with his great claws.', 'after\\r\\nthat he yawned until it seemed as if his jaws would crack, and then sat\\r\\ndown to think what he wanted for breakfast.', 'while he sat there, trying to make up his mind what would taste best, he\\r\\nwas listening to the sounds that told of the waking of all the little\\r\\npeople who live in the green forest.', 'he heard sammy jay way off in the\\r\\ndistance screaming, \"thief!', 'thief!\"', 'and grinned.', '\"i wonder,\" thought\\r\\nbuster, \"if some one has stolen sammy\\'s breakfast, or if he has stolen\\r\\nthe breakfast of some one else.', 'probably he is the thief himself.\"']\n"
     ]
    }
   ],
   "source": [
    "sents = nltk.sent_tokenize(burgess)\n",
    "print(sents[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[the adventures of buster bear by thornton w. burgess 1920]\\r\\n\\r\\ni\\r\\n\\r\\nbuster bear goes fishing\\r\\n\\r\\n\\r\\nbuster bear yawned as he lay on his comfortable bed of leaves and\\r\\nwatched the first early morning sunbeams creeping through the green\\r\\nforest to chase out the black shadows.'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the sentence tokenization basically works with ending punctuations such as periods and question marks. \n",
    "\n",
    "What if we want to remove the punctuations as well? In order to accomplish this, we can either come up with a list  unwanted punctuations or use a ready-to-use list in python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "def remove_punctuation(text):\n",
    "    text = \"\".join([word for word in text if word not in string.punctuation])\n",
    "    tokens = nltk.word_tokenize(text)\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the', 'adventures', 'of', 'buster', 'bear', 'by', 'thornton', 'w', 'burgess', '1920', 'i', 'buster', 'bear', 'goes', 'fishing', 'buster', 'bear', 'yawned', 'as', 'he', 'lay', 'on', 'his', 'comfortable', 'bed', 'of', 'leaves', 'and', 'watched', 'the', 'first', 'early', 'morning', 'sunbeams', 'creeping', 'through', 'the', 'green', 'forest', 'to', 'chase', 'out', 'the', 'black', 'shadows', 'once', 'more', 'he', 'yawned', 'and', 'slowly', 'got', 'to', 'his', 'feet', 'and', 'shook', 'himself', 'then', 'he', 'walked', 'over', 'to', 'a', 'big', 'pinetree', 'stood', 'up', 'on', 'his', 'hind', 'legs', 'reached', 'as', 'high', 'up', 'on', 'the', 'trunk', 'of', 'the', 'tree', 'as', 'he', 'could', 'and', 'scratched', 'the', 'bark', 'with', 'his', 'great', 'claws', 'after', 'that', 'he', 'yawned', 'until', 'it', 'seemed']\n"
     ]
    }
   ],
   "source": [
    "print(remove_punctuation(burgess)[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming and Lemmatization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For grammatical reasons, documents are going to use different forms of a word, such as organize, organizes, and organizing. \n",
    "\n",
    "Additionally, there are families of derivationally related words with similar meanings, such as democracy, democratic, and democratization. In many situations, it seems as if it would be useful for a search for one of these words to return documents that contain another word in the set.\n",
    "\n",
    "***Stemming*** usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes. (removing affixes from words and return the root word. Ex: The stem of the word working => work.)\n",
    "\n",
    "***Lemmatization*** usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma . \n",
    "\n",
    "If confronted with the token saw, stemming might return just s, whereas lemmatization would attempt to return either see or saw depending on whether the use of the token was as a verb or a noun. The two may also differ in that stemming most commonly collapses derivationally related words, whereas lemmatization commonly only collapses the different inflectional forms of a lemma. \n",
    "\n",
    "The goal of both stemming and lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. As we can understand, both techniques could remove important information but also help us to normalize our corpus (although lemmatization is the one that is usually applied)\n",
    "\n",
    "There are many algorithms for stemming and most of them produce similar results. Here are the embedded stemming algorithms in NLTK:\n",
    "\n",
    "- nltk.stem.ISRIStemmer\n",
    "\n",
    "- nltk.stem.LancasterStemmer\n",
    "\n",
    "- nltk.stem.PorterStemmer\n",
    "\n",
    "- nltk.stem.RegexpStemmer\n",
    "\n",
    "- nltk.stem.RSLPStemmer\n",
    "\n",
    "- nltk.stem.SnowballStemmer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "##### Porter Stemmer #####\n",
      "run\n",
      "runner\n",
      "decreas\n",
      "multiplying\n",
      "\n",
      "##### Lancaster Stemmer #####\n",
      "run\n",
      "run\n",
      "decreas\n",
      "multiplying\n",
      "\n",
      "##### Snowball Stemmer #####\n",
      "run\n",
      "runner\n",
      "decreas\n",
      "multiplying\n",
      "\n",
      "##### WordNet Lemmatizer #####\n",
      "running\n",
      "runner\n",
      "decrease\n",
      "multiplying\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#-----PorterStemmer------\n",
    "stemmer = nltk.stem.PorterStemmer()\n",
    "print('##### Porter Stemmer #####')\n",
    "print (stemmer.stem('running'))\n",
    "print (stemmer.stem('runner'))\n",
    "print (stemmer.stem('decreases'))\n",
    "print (stemmer.stem('multiplying\\n'))\n",
    "\n",
    "#-----LancasterStemmer------\n",
    "stemmer = nltk.stem.LancasterStemmer()\n",
    "print('##### Lancaster Stemmer #####')\n",
    "print (stemmer.stem('running'))\n",
    "print (stemmer.stem('runner'))\n",
    "print (stemmer.stem('decreases'))\n",
    "print (stemmer.stem('multiplying\\n'))\n",
    "\n",
    "#-----SnowballStemmer------\n",
    "# we need to specify language to initiate this stemmer\n",
    "stemmer = nltk.stem.SnowballStemmer(\"english\") \n",
    "print('##### Snowball Stemmer #####')\n",
    "print (stemmer.stem(\"running\"))\n",
    "print (stemmer.stem('runner'))\n",
    "print (stemmer.stem('decreases'))\n",
    "print (stemmer.stem('multiplying\\n'))\n",
    "\n",
    "#-----WordNetLemmatizer------\n",
    "lemmatizer = nltk.stem.WordNetLemmatizer()\n",
    "print('##### WordNet Lemmatizer #####')\n",
    "print(lemmatizer.lemmatize('running'))\n",
    "print (lemmatizer.lemmatize('runner'))\n",
    "print(lemmatizer.lemmatize('decreases'))\n",
    "print(lemmatizer.lemmatize('multiplying\\n'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stemming works on words without knowing its context and that’s why stemming has lower accuracy and faster than lemmatization. \n",
    "\n",
    "We can also specify the part-of-speech in lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "play\n",
      "playing\n",
      "playing\n",
      "playing\n"
     ]
    }
   ],
   "source": [
    "print(lemmatizer.lemmatize('playing', pos=\"v\"))\n",
    "print(lemmatizer.lemmatize('playing', pos=\"n\"))\n",
    "print(lemmatizer.lemmatize('playing', pos=\"a\"))\n",
    "print(lemmatizer.lemmatize('playing', pos=\"r\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part-of-Speech (POS) Tagging "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "POS Tagging is used when we need to analysis the words in the sentence for grammar and their arrangement in a manner that shows the relationships among the words.\n",
    "\n",
    "Apart from the grammar relations, every word in a sentence is also associated with a part of speech (pos) tag (nouns, verbs, adjectives, adverbs etc). The pos tags defines the usage and function of a word in the sentence. \n",
    "\n",
    "Part of Speech tagging is highly useful in terms of word sense disambiguation. The meaning of certain words would change based on the context. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "once more he yawned, and slowly\r\n",
      "got to his feet and shook himself.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('once', 'RB'),\n",
       " ('more', 'RBR'),\n",
       " ('he', 'PRP'),\n",
       " ('yawned', 'VBD'),\n",
       " (',', ','),\n",
       " ('and', 'CC'),\n",
       " ('slowly', 'RB'),\n",
       " ('got', 'VBD'),\n",
       " ('to', 'TO'),\n",
       " ('his', 'PRP$'),\n",
       " ('feet', 'NNS'),\n",
       " ('and', 'CC'),\n",
       " ('shook', 'NN'),\n",
       " ('himself', 'PRP'),\n",
       " ('.', '.')]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents = nltk.sent_tokenize(burgess)\n",
    "print (sents[1])\n",
    "tokens = nltk.word_tokenize(sents[1])\n",
    "nltk.pos_tag(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the meaning of POS tags:\n",
    "\n",
    "\n",
    "|Symbol| Explanation|Symbol| Explanation|\n",
    "|---|---|---|---|\n",
    "|CC | Coordinating conjunction | `PRP$` | Possessive pronoun |\n",
    "|CD | Cardinal number | RB | Adverb\n",
    "|DT | Determiner | RBR | Adverb, comparative |\n",
    "|EX | Existential there | RBS | Adverb, superlative |\n",
    "|FW | Foreign word | RP | Particle |\n",
    "|IN | Preposition or subordinating conjunction | SYM | Symbol |\n",
    "|JJ | Adjective |TO | to |\n",
    "|JJR | Adjective, comparative |UH | Interjection |\n",
    "|JJS | Adjective, superlative |VB | Verb, base form |\n",
    "|LS | List item marker |VBD | Verb, past tense |\n",
    "|MD | Modal |VBG | Verb, gerund or present participle |\n",
    "|NN | Noun, singular or mass |VBN | Verb, past participle |\n",
    "|NNS | Noun, plural |VBP | Verb, non-3rd person singular present |\n",
    "|NNP | Proper noun, singular |VBZ | Verb, 3rd person singular present |\n",
    "|NNPS | Proper noun, plural |WDT | Wh-determiner |\n",
    "|PDT | Predeterminer |WP | Wh-pronoun |\n",
    "|POS | Possessive ending |`WP$` | Possessive wh-pronoun |\n",
    "|PRP | Personal pronoun |WRB | Wh-adverb |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named Entity Recognition (NER)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the real world, in our daily conversations we don’t work directly with the categories of words. For example, if we want to build a Netflix chatbot we want it to recognize both ‘Joker’ and ‘Batman’ as instances of the same group which we call ‘movies’ , but ‘Christopher Nolen’ as a ‘director’. This concept of semantic field dependent of a context is what we define as entity. The role of a named entity recognizer is to detect relevant entities in our corpus.\n",
    "The process of detecting the named entities such as person names, location names, company names etc from the text is called as NER. For example :\n",
    "\n",
    "Sentence: Bill, the CEO of Microsoft Inc. is living in USA.\n",
    "\n",
    "Named Entities –  ( “person” : “Bill” ), (“org” : “Microsoft Inc.”), (“location” : “USA”)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Commonly Used Types of Named Entity***\n",
    "\n",
    "ORGANIZATION\tGeorgia-Pacific Corp., WHO\n",
    "\n",
    "PERSON\tEddy Bonte, President Obama\n",
    "\n",
    "LOCATION\tMurray River, Mount Everest\n",
    "\n",
    "DATE\tJune, 2008-06-29\n",
    "\n",
    "TIME\ttwo fifty a m, 1:30 p.m.\n",
    "\n",
    "MONEY\t175 million Canadian Dollars, GBP 10.40\n",
    "\n",
    "PERCENT\ttwenty pct, 18.75 %\n",
    "\n",
    "FACILITY\tWashington Monument, Stonehenge\n",
    "\n",
    "GPE\t(Geo-Political Entity) South East Asia, Midlothian"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The architecture for a simple information extraction system begins by processing a document using several of the procedures: \n",
    "- Raw text of the document is split into sentences using a sentence segmenter\n",
    "- Each sentence is further subdivided into words using a tokenizer. \n",
    "- Each sentence is tagged with part-of-speech tags\n",
    "- Named entity detection. \n",
    "- Finally, we use relation detection to search for likely relations between different entities in the text.\n",
    "\n",
    "We should keep in mind that because models are statistical and strongly depend on the examples they were trained on, this doesn't always work perfectly and might need some tuning later, depending on your use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/UAAABTCAIAAABRdLDAAAAJMmlDQ1BkZWZhdWx0X3JnYi5pY2MAAEiJlZVnUJNZF8fv8zzphUASQodQQ5EqJYCUEFoo0quoQOidUEVsiLgCK4qINEWQRQEXXJUia0UUC4uCAhZ0gywCyrpxFVFBWXDfGZ33HT+8/5l7z2/+c+bec8/5cAEgiINlwct7YlK6wNvJjhkYFMwE3yiMn5bC8fR0A9/VuxEArcR7ut/P+a4IEZFp/OW4uLxy+SmCdACg7GXWzEpPWeGjy0wPj//CZ1dYsFzgMt9Y4eh/eexLzr8s+pLj681dfhUKABwp+hsO/4b/c++KVDiC9NioyGymT3JUelaYIJKZttIJHpfL9BQkR8UmRH5T8P+V/B2lR2anr0RucsomQWx0TDrzfw41MjA0BF9n8cbrS48hRv9/z2dFX73kegDYcwAg+7564ZUAdO4CQPrRV09tua+UfAA67vAzBJn/eqiVDQ0IgALoQAYoAlWgCXSBETADlsAWOAAX4AF8QRDYAPggBiQCAcgCuWAHKABFYB84CKpALWgATaAVnAad4Dy4Aq6D2+AuGAaPgRBMgpdABN6BBQiCsBAZokEykBKkDulARhAbsoYcIDfIGwqCQqFoKAnKgHKhnVARVApVQXVQE/QLdA66At2EBqGH0Dg0A/0NfYQRmATTYQVYA9aH2TAHdoV94fVwNJwK58D58F64Aq6HT8Id8BX4NjwMC+GX8BwCECLCQJQRXYSNcBEPJBiJQgTIVqQQKUfqkVakG+lD7iFCZBb5gMKgaCgmShdliXJG+aH4qFTUVlQxqgp1AtWB6kXdQ42jRKjPaDJaHq2DtkDz0IHoaHQWugBdjm5Et6OvoYfRk+h3GAyGgWFhzDDOmCBMHGYzphhzGNOGuYwZxExg5rBYrAxWB2uF9cCGYdOxBdhK7EnsJewQdhL7HkfEKeGMcI64YFwSLg9XjmvGXcQN4aZwC3hxvDreAu+Bj8BvwpfgG/Dd+Dv4SfwCQYLAIlgRfAlxhB2ECkIr4RphjPCGSCSqEM2JXsRY4nZiBfEU8QZxnPiBRCVpk7ikEFIGaS/pOOky6SHpDZlM1iDbkoPJ6eS95CbyVfJT8nsxmpieGE8sQmybWLVYh9iQ2CsKnqJO4VA2UHIo5ZQzlDuUWXG8uIY4VzxMfKt4tfg58VHxOQmahKGEh0SiRLFEs8RNiWkqlqpBdaBGUPOpx6hXqRM0hKZK49L4tJ20Bto12iQdQ2fRefQ4ehH9Z/oAXSRJlTSW9JfMlqyWvCApZCAMDQaPkcAoYZxmjDA+SilIcaQipfZItUoNSc1Ly0nbSkdKF0q3SQ9Lf5RhyjjIxMvsl+mUeSKLktWW9ZLNkj0ie012Vo4uZynHlyuUOy33SB6W15b3lt8sf0y+X35OQVHBSSFFoVLhqsKsIkPRVjFOsUzxouKMEk3JWilWqUzpktILpiSTw0xgVjB7mSJleWVn5QzlOuUB5QUVloqfSp5Km8oTVYIqWzVKtUy1R1WkpqTmrpar1qL2SB2vzlaPUT+k3qc+r8HSCNDYrdGpMc2SZvFYOawW1pgmWdNGM1WzXvO+FkaLrRWvdVjrrjasbaIdo12tfUcH1jHVidU5rDO4Cr3KfFXSqvpVo7okXY5upm6L7rgeQ89NL0+vU++Vvpp+sP5+/T79zwYmBgkGDQaPDamGLoZ5ht2GfxtpG/GNqo3uryavdly9bXXX6tfGOsaRxkeMH5jQTNxNdpv0mHwyNTMVmLaazpipmYWa1ZiNsulsT3Yx+4Y52tzOfJv5efMPFqYW6RanLf6y1LWMt2y2nF7DWhO5pmHNhJWKVZhVnZXQmmkdan3UWmijbBNmU2/zzFbVNsK20XaKo8WJ45zkvLIzsBPYtdvNcy24W7iX7RF7J/tC+wEHqoOfQ5XDU0cVx2jHFkeRk4nTZqfLzmhnV+f9zqM8BR6f18QTuZi5bHHpdSW5+rhWuT5z03YTuHW7w+4u7gfcx9aqr01a2+kBPHgeBzyeeLI8Uz1/9cJ4eXpVez33NvTO9e7zofls9Gn2eedr51vi+9hP0y/Dr8ef4h/i3+Q/H2AfUBogDNQP3BJ4O0g2KDaoKxgb7B/cGDy3zmHdwXWTISYhBSEj61nrs9ff3CC7IWHDhY2UjWEbz4SiQwNCm0MXwzzC6sPmwnnhNeEiPpd/iP8ywjaiLGIm0iqyNHIqyiqqNGo62ir6QPRMjE1MecxsLDe2KvZ1nHNcbdx8vEf88filhICEtkRcYmjiuSRqUnxSb7JicnbyYIpOSkGKMNUi9WCqSOAqaEyD0tandaXTlz/F/gzNjF0Z45nWmdWZ77P8s85kS2QnZfdv0t60Z9NUjmPOT5tRm/mbe3KVc3fkjm/hbKnbCm0N39qzTXVb/rbJ7U7bT+wg7Ijf8VueQV5p3tudATu78xXyt+dP7HLa1VIgViAoGN1tubv2B9QPsT8M7Fm9p3LP58KIwltFBkXlRYvF/OJbPxr+WPHj0t6ovQMlpiVH9mH2Je0b2W+z/0SpRGlO6cQB9wMdZcyywrK3BzcevFluXF57iHAo45Cwwq2iq1Ktcl/lYlVM1XC1XXVbjXzNnpr5wxGHh47YHmmtVagtqv14NPbogzqnuo56jfryY5hjmceeN/g39P3E/qmpUbaxqPHT8aTjwhPeJ3qbzJqamuWbS1rgloyWmZMhJ+/+bP9zV6tua10bo63oFDiVcerFL6G/jJx2Pd1zhn2m9az62Zp2WnthB9SxqUPUGdMp7ArqGjzncq6n27K7/Ve9X4+fVz5ffUHyQslFwsX8i0uXci7NXU65PHsl+spEz8aex1cDr97v9eoduOZ67cZ1x+tX+zh9l25Y3Th/0+LmuVvsW523TW939Jv0t/9m8lv7gOlAxx2zO113ze92D64ZvDhkM3Tlnv296/d5928Prx0eHPEbeTAaMip8EPFg+mHCw9ePMh8tPN4+hh4rfCL+pPyp/NP637V+bxOaCi+M24/3P/N59niCP/Hyj7Q/Fifzn5Ofl08pTTVNG02fn3Gcufti3YvJlykvF2YL/pT4s+aV5quzf9n+1S8KFE2+Frxe+rv4jcyb42+N3/bMec49fZf4bmG+8L3M+xMf2B/6PgZ8nFrIWsQuVnzS+tT92fXz2FLi0tI/QiyQvpTNDAsAAAAJcEhZcwAADdcAAA3XAUIom3gAAAAddEVYdFNvZnR3YXJlAEdQTCBHaG9zdHNjcmlwdCA5LjI2WJButwAAIABJREFUeJztnU1sG1eW7y/1ZVNyLFGJ5FiYB8ml6Z6GhVl0KPdsjIkbIhdpwFmZbMzG01lYAqYHcBbdInejpdjdGweDAcheNOLsWMHbGJgsWA8vBuS3iFl5KwaDBCrJ/d6TLTvNkhXry5JVb3Gg21d1i6XiVxU//r+FQF1WXd6Pc889595Tt0KWZTEAAAAAAABAR9ATdAEAAAAAAAAADQP2PQAAAAAAAJ0D7HsAAAAAAAA6h76gCwAAAKDpqKpaLBaTyWQkElEUJejiAAAAaCJYvwcAgA4nlUqZpplOpzVNy2azQRcHAABAcwnh/BwAAOhsEomEqqr0WdO0WCwWbHkAAAA0Fdj3AADQ4ei6ns1mI5HI7OxsIpEIujgAAACaC+x7AADoFigKP5PJBF0QAAAATQTx9wAA0OGkUin6kEgkTNMMtjAAAACaDc7PAQCADkfTNDLxTdOMx+NBFwcAAEBzQXwOAAB0PqZp6rqOJ2sBAKAbgH0PAAAAAABA54D4ewAAAAAAADoHxN8DAEAnYzx//t+Lxf/5X//17eZmZGjoyjvvvP+Tn/zTP/xDZGgo6KIBAABoCojPAQCAlsN4/tx48YL/+8P+/tdPnnz77Bn9u/ny5bPtbcaYubOz9/r1m+Pjw+Nji7Hj4+PjOlR6f28vfQj3958fGGCMRQYHGWNT77wzHA7TVxfD4Z9OTv740iXxxujUFLwFAABoHWDfAwBAY7AZ5Ywxc3e3uLZGn7d2d7d2dxlj321uMsYOj49/2NtjjB0cHf3lhx8Oj4+bUaSeUCgUCp3v6xsZGipvb1uh0MVweHt///XRUT2ewJmc6+t7d3iYPocHBsL9/YyxH514BVPvvEMfpsfHlbEx272xmZnmFQwAALoB2PcAgK5GX183d3bEFOPFi9Xnz+2XPXnCGNve2zt684ZSvt3c3Dk48KeQnFAo1NfTY1nW0Wl/4N3h4cvDw3//N3/z7vAwGc3ymnoqn//dF19Yf/qTmKiVSuSE0F+5UkPnzr0+Ojo8qXUgUO34v6MXLjDGopOTtsvgLQAAAIH4ewBA+1GVUS7yP775prklqwytYe8dHla64N3h4eFweHBg4I1lne/r29zetizrz+WyeI1lWf/4d38XGRxUxsZGL1yITk5GhoaiU1NeCkBmsb6+Ll5P5m/i2jXxStHoN1680NfXaeeB85PLly3LGg6Hj46PLcvq7+0tbWxU8nYGBwZ6enoODg/PdBJ6QqHwwMDuwYFt2enZy5fPXr60XVxzV06+/fbfno4vYvAWAACdBdbvAQA+oZVKtpRmG+WRoaGjN29+2N/3fktfb++Rh7Xqty9c6A2F6OK+3t7+3t5nL18eHB1Vun7u6lX6QHbk1u7u+MWLz16+3N7bOzg6enVwINvQ75HtPjnJGIvPzLC6jUutVIr/4Q/5f/kXmzXvEfKpCqUSY0x/8sR48WLtdDDS3NWrkcHBwYGB/r6+H1+6tPr8+cjgIBM61KUfJ99+mzHWEwpR1FB/b++rgwPZprdBUUDb+/uMsf6enp6enuPjY3Nnp6pgp/P9/f29vVUJiciVsTHZDYC3AAAIFtj3AAA3ZKNcf/Kk/OqVmEKrvKdSdna+lsx0j0y+/fbhmzfiOvfx8fHLvT3vOfC4FI/G/ZWxscvDw3uvXzPGzvX18cD0ze3t0aEhxtj//vOfXe4lu00ZG6OnUWlZnYqhjI3p6+viWrhsFlMOdHul0JqGYO7sjP7rvy5+8EEmmWxUnmca/dxLoWZRxsYiQ0P6+jo7/XAC+QDy7RweokPxQoyxyyMjh2/ePN3auhgO9/X2utzLGTp37seXLpV3dhhjw4OD5/v6GGMv9/ZCoRDtrjDGvn/16v+c3jNxZ3Bg4NzJvYyx/p6eUCi0ub3tPQcR2VugvRrbZbNXrpCw/fUyz9s4AIBuAPY9AB2Iz0a5aDZV4slf/uI9Q/HpzO39/f6envDAgPdM+GI5Yyw6Ofns5cv9w0MmrM2/OjjY2t0NDwy4G4W2RXcm2FXK2JgyPi5eTG3OLV3HxqQV7hpCaxpF5Ne/jl29qv761039FZvRLzeFzeiX24E/qczl1njxwtzdpcxtGx22bBlj02NjdE10aurp1tbG1hY90SvKvEs+nCtjY/9tdJQeg748MnLh3DnG2Pr330eGhviBQj/s778WNny8uBkioqgzxvZevyZR5/T09FSVoQhvEA68BQC6BNj3ALQE5s4OLWqKFCQzvYFG+cjg4NWJCVq0Jg6PjweFlUjG2Ob29sVwuK+nhzH26uCADn7xyN+Oj791/jxjTFwufbm3R6Z2eGDg/5bLrzw8nyraKGLYw/T4+Nhbb1GRLMvi3oWXaJCRwUEyX7i5wxfd2VlRE2R6kt1JRqcPoTWNIvb735s7O/rSkv8/Lbabo9FPS9e80c40MbkTy4cJdb3LiJD7nTZMvn327K3z5y+PjLDTnjB3Kphnf4Bbz3w/Z/377ydGRihzxthfXN3saoczudaMsb3DQxrLly5e5N9ubm/zw4uI59vb/29ry3v+IvAWAGgjYN8DUDsejXLRSuApta3Jydv323t7V05OG2SMbW5v771+fXlk5FxfH2Ps1cHB060tRmcUnrVcLUML2Nzc+dGlS5Q//ftWOMy/eur0BKR7+bkBxE6b14yx2MwMf4KWx+hzS8i9FtwK4Rny0Gd50d0F6twWCa1pFI5H6ARIw41+ET48qxIhLqLypg3vXPHx7ob7A+x0sP7TrS3uGzBpI052+Kt9WGXu6lUxQzrDlJ/lKu5UfLe52d/b+9bJv6wOPcbgLQDQZGDfg65DPqSc+W6UizPZ/uEhWcZ80uI+w7vDw+f7+23Pm1Y1f/MFS/67gwMD+yeh7YfHx/09Pfzb7zY36YgVj4uIYua2uTkuLFfbTF7RKbKtvDJXq0j8OW578R+qZ4G8LUJrGkLmP/8zrarFf/u3Vi48jVAyymkY2mReNPrJ/qu597mlLgcCuQw0OXCL5LCSVDTDHxD9YdEItg0320lT8hPt9agXJpnpytjYS6EWb4XDP750iUKkKIW/+kB+jL4eb0EMyeMlsTkG7LRe4pd59/kBaCNg34P2wP3NQRx5zvAyazri8UE3mjCebm09PVm9/tGlS0fHx7ayiQWrdgtenkH5vLW1uzs4MEDLez/s73/77NnI4KDt0BLmuRFsYev8s2g9uE+HvJtEM+LMqAmxjrx2fAmzgZZ0W4fWNIQ6j9AJEHILXYx+cv9Ifuo0+m3UEwgky7MXg7KSPyAOai+GeKXYNhd/wKUkzIN7UK2+dVFujLHRCxdC0vUeD92qWfMzeAugI4B9D5qI/0a5vOcr62VbKAgRnZoyXrwQZzL3ffDaAl3++nOnC0DT7Q/7+xRN/qNLl946f17cUhDX+Tz+tNgUlbb+qzKdZSvHyyOPzPUp1eYdFNN5oTUNoRlH6ASIaPQ7HtUvG/3N6GXZreXqwsvTwPK4qNYzEZ+nF/WG//6ADdsUIOv/Ot0D2yqMvAQjmuB0lBY9FCQWwHbwgFyqGgomAm8BBALse2Cn5jcH+W+U22ZBORreFnXT2EAXcSKRixebmdFOT7R8FhFdBY9r+eI0Jv60HLbuvUYcPgfL5xW6F8/9aEh/IkC6J7SmUfhzhE6wnPl+Li4Svrl2ciCQlwfBbb5xo8ZXy/oDNs50D2xRlNUuu1TlHrDTBrfjw1ct4i1UWsPq+PULIAL7vkPw/3WeNWsZL4eT8H/d13saGOjCpBfQ2NZOxILZ2rbaSdHmKohN5BK2XhuaEFFgCy+u+WjIQOYJhNY0igCP0AmWFjT65RKyKl8LIPvYXI81sPyN8gcqRQCKeq+BY/bMFZ863QObJmcVNmb5vy7r8YF4C3L5mdOr2eAttCmw74PE/9d5yka5l/csVrtWZHM2/A90+eu3FXSQqExtLkQDw9bFlmz4TiuvgmwNMFcJ4QZBDUdD+gBCa5pNqx2hEyy2/Z9KL+Xlmz/BhkzIgUBVvRagsU+lu8Cntkra1ePSjM/+gI0z3YM6zzat1j3wPhF79Bbkw5dYHdYFvIVWA/Z91fj/Ok9/jHKRVgt0OTPPSstLDQxbFwvW7AAPeSvf56Mh/QGhNYHQFkfoBEsNL+VtncHFtYf3p4HZySwj+/w+VM3Laktb+AMymjRvuix11WAkuK922ayCmpWnxxPn4C20Gt1i3/tslHsUzaae7NuygS5nIi7/ixsaDQxbZwFp/PqPhhRr0exFuEaB0JqWon2P0AmWal/K21JGv4i89VfVawG4tg8qYK/Z/oCfSznuaFI0UWPPNq3KPWAN0smt7C3IVWbtPBO1rn0fyOs8azPKm6fHWz/Q5UzEwewSCVNn2HqjHueqB5czNFrhaEgfQGhNW9BhR+gES1PfzxUgXJvJz+3UEAgUbK0b5Q9Uet1H6/gDNvx/9YHtbNMatuKrJRBvQT47m7Wkt+CrfZ/K5+lDU98cxII2yr2glUp8HUhMDzzQpQaoW5sUtt5S6pKgvqv5aEi+6N4Zdm0qn0doTdvRDUfoBIh3o3/hxo3WXOP3Qj2vBSDN0JrVr3SUgmgjerFYZH9genx8/saNZpS5Gdjcg4afbVrJPQhQKjye6C1bsPUEYLsbsfW0hq/2fXRpiZqgqjcHncqhI0widvKIG2tyoIs/hD76iHkLW2ftvNXFUR8/Tv7Hf7TC0ZCtQOijjxBa03bkvvwyOjXVPVLaCji+lLf87//eGTOaI3zhXN7SJPu4M6rv3R+Yu3pV++1vgyml79T86oPCb37TGTOIfKphDd5CPa3RuvE5AAAAAAAAgGrpCboAAAAAAAAAgIbRV8M9hmEYhkGfFUVRFEVMYYxFIpFoNCpfxi9QVbVYLCaTyUgkwtMpMR6Px2IxXddN04zFYowxTdPkHNoITdMKhcL09HQ0Go1EInKbOLZeMGVtT0TJYU7ySZ/FjqAWJtESG7yS4FG6mKeYQj3rmKFpmrqui6WlzOUM5Yq01CgwTXN5eZk+ZDIZqq+MrV6mabZOFVoZjzLMJDGuR4bZ6f4iMa5ThlnnanJ3NE1bXl6mOlZ7IzvdRzs7O0MnUStyz9JnrnOaCgkbYyyTydjSvVS25jZxzIqdbqVIJCLOp7Z5U9b2zaPmalaaVrphvDCpjxyVHqyjeqhx/T6VStGHbDarqqqYIn6WL6NE0zTT6bSmadls1pao63oul2OMxeNxfks+n/dBlzWDVCplGEY6nY5EIolEglVuE/EW/8vZvsiSwyrIp60jGGORSKRQKCwvL4saxEXwHLspn8+Lc4wtQ13XyTai6/Mnj5g7ZihXpHVGQS6Xi8fjmUwml8udWQyxXq1ThZbFowwzJzGuR4aZkxjXI8OOdekSAYhGozYj2COGYYhNSnMib1VN0+RpQtQ5TSUWi2UyGZt3xzxXtuY2kXFsJVa5oWRt3zzqrKZtBHXJeHHsI1hHDcaqibm5Odtn+ru6umpZVjablS+7deuW7YNlWYVCwbKsYrHIb7Esa3FxsVwuz83N3blzp1wuU0pt5QyWYrEolrxYLFpOTWdVaD1wJo6SY0mC59gRxJ07d1ZXV5eXl3mKi+CJ2dJXtpzlDMvlslgk6mI5Q5eK+DwKqEYE/a5lWdlslkqyuLhIY9Ydm5C3+0BuKh5l2KqgT6w6ZNiqIMa1yXClunSDABQKBRoyYqI4lPL5vMvtfE4sl8uUiePUWUnnNBubzNgqm81mb926Rf2+uLh469YtUhFym6yurt66dWt5eXlxcZGLhGVZ+XyeUvL5/J07d0T9LCK3klXNMGkS3qtZCVvzdsN4qdRHsI4aSy3xOQRtHhUKhYWFBUpJpVK6rmuaNj8/L15mGEahUEin05SSTqfn5+cjkcjs7Cz5bfl8nmfCGJudnaUFg3Q6vby83KgFAP8xTTMej/N/+b6S3HSsQusBdxwlh29ucsGr1BGGYczOziqKsrq6KmbrRfAymYxpmtlsli+4OmZoW32ptNPqUhE/RwEF4dDCiWmaqVSKajc/P7+6usqjR6ql3QdyU/Eow6yCPqlHhpmTGNcsw5Xq4r0w7UvsBDExm81SJJuu6+7L7fF4XNO0WCyWy+XE+ZQxpus6nzoddY7/2CpLyoGm8ng8bhgGfSW3CQXsxWKxaDRK2zu0iFsoFKhG1FyVAjAcW4lJDVVJ2zcJj9WsKs+OHy8ufQTrqIHU/nxtoVAoFArJZJLvrWQymWQyyRjjIWUcRVF4F0aj0VwuR7JLcj86OireQqF17GQikTcH2w5VVWOxGI9tlZuOubYeqEQlyeGIgsekjshms9lsNhaLaZomRk96FLzl5WWaUfi9lTKspyJ+jgJd15Mn7zyi8NaGZNsxA7kZVCvD7LQY1ynDTBLjmmXYpS7dKQDpdDqVSqVSqXw+7+4YJxIJCnMvl8vcfcpkMhQOR8+9ELLOaQUWFhaokIVC4cx4GBLmaDRaLpcZY4ZhcJ3jfq9jK7EKDcUkbe8ztmpWRfeMF7mPYB01kNrtexpUtomHvCuxY2Kx2Pz8fDwel4PyE4kE9VYikRBD64rFIs82k8nwSLu2Q1EU0keJRELTtGg0SlresemYU+u1FPPz8x7XIbxfWScukiMKXqWOYIzRqoCu63QB50zBU1V1enqaFLE4r1TKsOaKeClMo1AUpVgs8n8bGOPb1gO5qXiUYVZZn9Qsw6yCGNcmw+516UIBUFWVVrJoy9rlSvKCcrnc7Oys7Svy7mgkVtI5gUMr1vz5jWrv5TLDQ64dcWklJjSUi7ZvIzp7vLj0UZtaR/VjmqbHfTnvV/YuLS1VWw5N0+7fv7+xsbG2tsbPIaEUWvJ59OjR3bt3eWI8HlcURdM0uj6dThuGQU/DxOPxmZmZSCSytraWz+cpcWFh4Ztvvrl//344HI5Go+FwWFXV27dvV1vOwBHrpWna/v7+6OiorelYhdYLtuQypmn+8pe/XFpaOvNZfu9X1o8sORMTE7LgbW1tjY6Oih2RTCbn5+efPn06OjqqKMpnn32mqmooFDJN01HwcrlcPp9/9OjRxsZGKBRSFOXmzZuXL1+mDJ8+fXr79m3HDGmL9t69e3Tv7OxsOByWM4xGo5Uq4ucooPbMZrMrKyu5XC6dTk9MTFBpNU3b2NhYWVmh6lfKwVYvwzA6YCA3FY8yvLa2FovFbPqEZK9mGWaM2cR4ZWWlZhlWFKVTNfmZpFIpUt2ibl9aWiqVSpqmPXjw4MMPP5xxfUnN6Ojo0tLSvXv32OkZQVXVcDhMi5eyzgmkXo6J4XA4kUh89tlnJBuOl4kKjVTKzZs3JyYm9vf37927t7KyMjMzQzLvpZUqNZQ87fL9AX+ayLGavFlE5BHkv84PBMc+kg1L1ibWUUP45JNPPv74YxoRjboymPdb0YFrNq/aMbEz0DSt3U+5Ik3kZaPT+5WNwrvktHhHtMgQaJFidBVVtTnEuF1AU1SFYRiqqjZq77fFhwlg6KPT6Lru8XERj1fi/bXAE97HIUYsAAAA75BNbxiGl+N3AQBegH0PAAAAAABA51D7+ZgAdC36+np0airoUgBQC8bz52qx+L++++4nly8nf/YzSDIAoHUwd3bMnR1lfDzogrQ9vq7fx37/e8aY9tvf+vaLLUsqn//dF19Yf/pT0AUBVaOVSvE//KHwm9/EXB+YA6ClMHd21MePC6XS58UiY+zSxYub29uMsfcmJ5M/+1lidhYTKgAgcEIffbT4wQeZZj4V3SVg/R4AADoZMuv/+PAhY2xkcPDO++8v3LgRnZqihfzsl1+mVTWtqjD0AQCgY4B9DwAAHQiZ9erjx1u7u2TWx2dmEteu8QuU8fHUL36R+sUv9PX1/FdfqcUiGfq3ZmfpysjQUIDlBwAAUDOw7wEAoHPgxvraixeMMS/GenRqKjo1lUkm6d7cw4efF4sLn35K987fuOFf6QEA3c3I4GDQRegQYN8DAEDbw4NtyKx/b3Iy/c//XO0aPDf0+dr/58ViSlUT167Z1v4BAKAZRKem9CdPgi5FJwD7HgAA2hUy6/NfffX1kyeMsfcmJxcSifpj6BPXriWuXcv96lfq48f5r77648OHf3z4cGRwMHHtWvLaNTxZDgAALQ7sewAAaDNsh+FcGRtb/OCDZhx2SYY+/zky9K+MjSVmZ3G2JgAAtCyw7wGoDmVsjDFmvHgRdEFAN1LpMJym/mhkaGj+xo35GzfI0M9++eXvvvjid198QYb+wo0bOHIHAABaCtj3AFQHmTKrz58HXRDQRZx5GI4/cEOfxwWRoY+zNQEAoKWAfQ8AAC1KDYfh+AM/W9N4/jz75Zf8bM33JicXbtxohRICAEA3A/seAABai4YchuMPyvh4JpnkZ2uqxeLCp5/yszVbs8wAANDxwL4HAICWoEmH4fgDP1tTK5Xyjx/T2ZoLn34aVCgRAKAdiU5O5h4+DLoUnQDsewAACBLfDsPxh9jMTGxmhs7WpGcG+NmaMPQBAGeytbsbdBE6Adj3AAAQDIEchuMbdLZmJpEQz9YcGRycf//99vVeAACgLYB9DwAAvtIih+H4g+1szUKpJJ6tCUMfAACaAex7AKpm7upVvEAbVEvLHobjDzhbEwAAfAP2PQAANJE2OgzHH8SzNall+NmaMPQBAKAhwL4HAIDG09aH4fgDN/T5zgYZ+t22swEAAA3HV/s+Ojnp58+1MtPj43NXrwZdClA7EGbgTkpVPy8W2/0wHH/gZ2uSoZ97+PDzYrG4vp771a+CLhoAwFdGL1y4MjYWdCk6gZBlWUGXAQAAOg2tVIoMDcGsrw318WNlbAytBwAAtQH7HgAAAAAAgM7hjPgcwzAMw6DPkUgkGo3yrzRNW15e1jSN/1soFBhjmUxGvEZOlNF13TTNWCxGtzDGdnZ2hoaGxBRFUag8dIuiKJTiA1SL6enpaDQqtoANwzAikUgkEmnIj8ptoiiKaZq2xL6+vqOjI7rFzzZpa1RVLRaL8XicRFoUcnZazvmV1OYyHrspQOntTnRdZ4zxftQ0jbrVprX8xDTN5eVl+pDJZCKRiBeN0V4CRqVtahmgGAFoHrbJ0fGaSnado3aV9V6dJaxNh3en3ug584pUKkUfNE1TVZWnR6NRsXdjsVgmk6Fp1T3RkXg8zjPP5/MXL160pZBY8MJks1mxMM0jlUoZhpFOpyORSCKRcLkym816qal3HFvAljg8POx/m7Q1qVTKNM10Om0YBrfaeRuKn/mVuq7ncrlKGXrppqCkt2vRdZ1GLmNM07T5+XnqApvW8pNcLhePxzOZTC6Xo8J41BhtJ2DiaGoGUIwANAPHyVGmkl3nqF1lvVcnNevwLtQbZ6zfK4oSiUSop2OxWCKRIBuXO3Au69neiUajc3NzhUIhFovFYrFCofDzn//clhI5gReGylP/r7tAEjw/P88YSyQS3Jnj1WcnLqymaeQg2vxaTdO4JJGToOt6Pp+nb2dnZytVQW4T8qdtiT/96U99bpO2Rtf10dFR6lD6ywQhNwxDURQyCnVdn56epmtSqRQpPlk9eeymQKS3m1EUJZlMZrPZTCaTz+dp8DpqLcMwstks/5dGbi6XKxQK6XQ6n8/zZSdx5LqMZfkynuHq6mqhUKDtoEoaw0Z7CRg1LC8GY8wwjFQqNTs7Wy6XxQU8x2b3+BNQjAA0HMfJUTZ1KuGoXWW956IeRX1rmqaj3pB/xWMJu1RvWGcxNze3uLi4uLhIDWH7Sr7YMQcvv7K6urq4uGhZFv2VUyixUCgUCoXFxcV8Pn9mtnVCv+VyQT6f5xcsLi7aLublFz/fuXOnXC5bllUsFt2rUKkF5Ibys03amkodSkIuCuri4uLq6ir/V+xo+V4v3WShp3yEt3OxWFxeXuZdYEnqiI9HS+gp60QkLMuib8vl8q1bt+ircrl8584dy2ksO17GM7eJkJziSNsJmK2F5+bmisWiZVnUF5RYqdk95g/FCEBjcbd25BnQo7EnajkX9WjTt1YFveHy0y5zNL+r2/SGp/MxySsyDGN+ft5l16ZOaHVc3PSRUxhj5Kslk8mGbB14RFVV2kxXVTUWi9HWP631LiwsVLrLMAxyQ+lfikxNp9OpVIqv6Lv/rmMLyImBtEmHQRuIjDFapx8dHeWRxJToEornsZsYespfFhYWaMWIAkAdESPgbcs/9C99q+t6Mpm03SKPZcfL6qfdBYwKE41G+dKdS7N7AYoRAB/waOp4xF09ivqWkPVGnSXsNr1Rxfn3iqLQw4jNe+Ygk8nwSFmXlCb9uoyiKNlslgcm0X4NRW5QMdzDsyjww1ZgVVW5HTk/P39mgJfcAnJiUCHFbUc0Gk2lUtxHVVVV3H3jgViapiUSiWw2y4d3sVjkW5aOeOkmhp7yFx5t5YLoxbkoN1IFXFooW3ksO17WEDpMwByb3TRNVVXdBxoHihGABiJPjoqieDR1PNJw9ejdGON0ld7oXVpacvla07T79+9vbGzQw7XhcJjcr1QqpWnao0ePNjY21tbWSDJsKY6XufxKOByORqPhcFhV1YmJCVvK7du3eWFcsmoskUhkbW0tn89rmqZp2v7+fjKZDIfD9+7d03Vd07RSqVQqla5fvx4Oh6enp7PZ7MrKiqZpVEK6nSd+/fXX169fX1paKpVKmqY9ePDgww8/nJmZ8dgmvAXkhvKzTdqacDg8MTFx9+5d6r6NjY14PC4KOYnr3bt3xa5XVXVhYWFiYkLO0GM3BSK93czy8nKpVAqHw4qiqKr64MGD/f196k2bOnr69Ckf4Lqux+NxxlgqlaLLNE2bnp6mpSY+lnO5XDqdnpiYkMey42WMsVwuR/K2srISCoXInJU1hlyR9hKwXC6Xz+ep6aiaYlGpEW7evBkOhx2b/ZNPPvn4449v3rzpONYIKEYAmoE8Od68edPR1PFu7Nn0XjQadVSPsr6tpDdsv3L9+nXHEsq16069gfPvPaFpmu3BiJ5JAAAAhElEQVSwJE3TyIIXLzNNU9d1WwiTnOh4GfATx+6TQU91CTXLg8ch7z3DrkJudl3XO2BaBaB9sY1Kj7rROw1Xeg0vYccA+x4AAAAAAIDO4ezz7wEAAAAAAADtAux7AAAAAAAAOgfY9wAAAAAAAHQOsO8BAAAAAADoHGDfAwAAAAAA0Dn8f07J189hvEnqAAAAAElFTkSuQmCC",
      "text/plain": [
       "Tree('S', [Tree('PERSON', [('Bill', 'NNP')]), Tree('GPE', [('Gates', 'NNP')]), (',', ','), Tree('ORGANIZATION', [('CEO', 'NNP')]), ('of', 'IN'), Tree('ORGANIZATION', [('Microsoft', 'NNP'), ('Inc.', 'NNP')]), ('is', 'VBZ'), ('living', 'VBG'), ('in', 'IN'), Tree('GPE', [('California', 'NNP')]), ('.', '.')])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = \"Bill Gates, CEO of Microsoft Inc. is living in California.\"\n",
    "\n",
    "tokens = nltk.word_tokenize(sentence)\n",
    "chunks = nltk.ne_chunk(nltk.pos_tag(tokens))\n",
    "chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(PERSON Bill/NNP)\n",
      "(GPE Gates/NNP)\n",
      "(',', ',')\n",
      "(ORGANIZATION CEO/NNP)\n",
      "('of', 'IN')\n",
      "(ORGANIZATION Microsoft/NNP Inc./NNP)\n",
      "('is', 'VBZ')\n",
      "('living', 'VBG')\n",
      "('in', 'IN')\n",
      "(GPE California/NNP)\n",
      "('.', '.')\n"
     ]
    }
   ],
   "source": [
    "for chunk in chunks:\n",
    "    print(chunk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PERSON Bill\n",
      "GPE Gates\n",
      "ORGANIZATION CEO\n",
      "ORGANIZATION Microsoft Inc.\n",
      "GPE California\n"
     ]
    }
   ],
   "source": [
    "for chunk in chunks:\n",
    "    if hasattr(chunk, 'label'):\n",
    "        print(chunk.label(), ' '.join(c[0] for c in chunk))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can realize, Bill is recognizd as name but CEO is recognized as organization. Since we used the simplest way of NER eembedded in NLTK, it's not strong enough to catch these kind of consecutive entities as a one. In practice, we generally use much stronger and efficient NER packages such as StandfordNERTagger, Spacy etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLP Techniques and Algorithms\n",
    "\n",
    "### N-grams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N-gram is a continuous sequence of n items from a given sequence of text or speech and N-grams of texts are extensively used in text mining and natural language processing tasks. In other words, n-grams are simply all combinations of adjacent words or letters of length n that you can find in your source text. If it's a one word gram, it's called unigram, bigram for two-word grams, trigram for three-word grams and so on. They are basically a set of co-occuring words within a given window and when computing the n-grams you typically move one word forward.\n",
    "\n",
    "N-grams are used for a variety of different task. For example, when developing a language model, n-grams are used to develop not just unigram models but also bigram and trigram models. Another simple purpose would be building a keyword sequesnces made of multiple words. One another use of n-grams, may be the most important one, is for developing features for supervised Machine Learning models such as SVMs, Naive Bayes, etc. The idea is to use tokens such as bigrams in the feature space instead of just unigrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('the', 'adventures'), ('adventures', 'of'), ('of', 'buster'), ('buster', 'bear'), ('bear', 'by'), ('by', 'thornton'), ('thornton', 'w'), ('w', 'burgess'), ('burgess', '1920'), ('1920', 'i'), ('i', 'buster'), ('buster', 'bear'), ('bear', 'goes'), ('goes', 'fishing'), ('fishing', 'buster'), ('buster', 'bear'), ('bear', 'yawned'), ('yawned', 'as'), ('as', 'he'), ('he', 'lay'), ('lay', 'on'), ('on', 'his'), ('his', 'comfortable'), ('comfortable', 'bed'), ('bed', 'of'), ('of', 'leaves'), ('leaves', 'and'), ('and', 'watched'), ('watched', 'the'), ('the', 'first'), ('first', 'early'), ('early', 'morning'), ('morning', 'sunbeams'), ('sunbeams', 'creeping'), ('creeping', 'through'), ('through', 'the'), ('the', 'green'), ('green', 'forest'), ('forest', 'to'), ('to', 'chase'), ('chase', 'out'), ('out', 'the'), ('the', 'black'), ('black', 'shadows'), ('shadows', 'once'), ('once', 'more'), ('more', 'he'), ('he', 'yawned'), ('yawned', 'and'), ('and', 'slowly')]\n"
     ]
    }
   ],
   "source": [
    "tokens = remove_punctuation(burgess)\n",
    "bigrams = nltk.bigrams(tokens)\n",
    "print(list(bigrams)[:50])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('the', 'adventures', 'of'), ('adventures', 'of', 'buster'), ('of', 'buster', 'bear'), ('buster', 'bear', 'by'), ('bear', 'by', 'thornton'), ('by', 'thornton', 'w'), ('thornton', 'w', 'burgess'), ('w', 'burgess', '1920'), ('burgess', '1920', 'i'), ('1920', 'i', 'buster'), ('i', 'buster', 'bear'), ('buster', 'bear', 'goes'), ('bear', 'goes', 'fishing'), ('goes', 'fishing', 'buster'), ('fishing', 'buster', 'bear'), ('buster', 'bear', 'yawned'), ('bear', 'yawned', 'as'), ('yawned', 'as', 'he'), ('as', 'he', 'lay'), ('he', 'lay', 'on'), ('lay', 'on', 'his'), ('on', 'his', 'comfortable'), ('his', 'comfortable', 'bed'), ('comfortable', 'bed', 'of'), ('bed', 'of', 'leaves'), ('of', 'leaves', 'and'), ('leaves', 'and', 'watched'), ('and', 'watched', 'the'), ('watched', 'the', 'first'), ('the', 'first', 'early'), ('first', 'early', 'morning'), ('early', 'morning', 'sunbeams'), ('morning', 'sunbeams', 'creeping'), ('sunbeams', 'creeping', 'through'), ('creeping', 'through', 'the'), ('through', 'the', 'green'), ('the', 'green', 'forest'), ('green', 'forest', 'to'), ('forest', 'to', 'chase'), ('to', 'chase', 'out'), ('chase', 'out', 'the'), ('out', 'the', 'black'), ('the', 'black', 'shadows'), ('black', 'shadows', 'once'), ('shadows', 'once', 'more'), ('once', 'more', 'he'), ('more', 'he', 'yawned'), ('he', 'yawned', 'and'), ('yawned', 'and', 'slowly'), ('and', 'slowly', 'got')]\n"
     ]
    }
   ],
   "source": [
    "trigrams = nltk.trigrams(tokens)\n",
    "print(list(trigrams)[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the ngrams larger than three, we can use another nltk function with the number of grams as a parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('the', 'adventures', 'of', 'buster'), ('adventures', 'of', 'buster', 'bear'), ('of', 'buster', 'bear', 'by'), ('buster', 'bear', 'by', 'thornton'), ('bear', 'by', 'thornton', 'w'), ('by', 'thornton', 'w', 'burgess'), ('thornton', 'w', 'burgess', '1920'), ('w', 'burgess', '1920', 'i'), ('burgess', '1920', 'i', 'buster'), ('1920', 'i', 'buster', 'bear'), ('i', 'buster', 'bear', 'goes'), ('buster', 'bear', 'goes', 'fishing'), ('bear', 'goes', 'fishing', 'buster'), ('goes', 'fishing', 'buster', 'bear'), ('fishing', 'buster', 'bear', 'yawned'), ('buster', 'bear', 'yawned', 'as'), ('bear', 'yawned', 'as', 'he'), ('yawned', 'as', 'he', 'lay'), ('as', 'he', 'lay', 'on'), ('he', 'lay', 'on', 'his'), ('lay', 'on', 'his', 'comfortable'), ('on', 'his', 'comfortable', 'bed'), ('his', 'comfortable', 'bed', 'of'), ('comfortable', 'bed', 'of', 'leaves'), ('bed', 'of', 'leaves', 'and'), ('of', 'leaves', 'and', 'watched'), ('leaves', 'and', 'watched', 'the'), ('and', 'watched', 'the', 'first'), ('watched', 'the', 'first', 'early'), ('the', 'first', 'early', 'morning'), ('first', 'early', 'morning', 'sunbeams'), ('early', 'morning', 'sunbeams', 'creeping'), ('morning', 'sunbeams', 'creeping', 'through'), ('sunbeams', 'creeping', 'through', 'the'), ('creeping', 'through', 'the', 'green'), ('through', 'the', 'green', 'forest'), ('the', 'green', 'forest', 'to'), ('green', 'forest', 'to', 'chase'), ('forest', 'to', 'chase', 'out'), ('to', 'chase', 'out', 'the'), ('chase', 'out', 'the', 'black'), ('out', 'the', 'black', 'shadows'), ('the', 'black', 'shadows', 'once'), ('black', 'shadows', 'once', 'more'), ('shadows', 'once', 'more', 'he'), ('once', 'more', 'he', 'yawned'), ('more', 'he', 'yawned', 'and'), ('he', 'yawned', 'and', 'slowly'), ('yawned', 'and', 'slowly', 'got'), ('and', 'slowly', 'got', 'to')]\n"
     ]
    }
   ],
   "source": [
    "fourgrams = nltk.ngrams(tokens, 4)\n",
    "print(list(fourgrams)[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bag of Words - Term Frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A problem with modeling text is that it is messy, and techniques like machine learning algorithms prefer well defined fixed-length inputs and outputs. Machine learning algorithms cannot work with raw text directly; so the text must be converted into numbers. Specifically, vectors of numbers. If we want to use text in machine learning algorithms, we’ll have to convert them to a numerical representation. One of the methods is called bag-of-words approach.\n",
    "\n",
    "The bag of words model , or BoW for short, is a way of extracting features from text for use in modeling and it ignores grammar and order of words. Once we have a corpus (text data) then first, a list of vocabulary is created based on the entire corpus. Then each document or data entry is represented as numerical vectors based on the vocabulary built from the corpora."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['working', 'Data', 'statistics', 'as', 'studying', 'I', 'AI', 'love', 'Scientist']\n"
     ]
    }
   ],
   "source": [
    "sentence_1 = \"I love studying AI\"\n",
    "sentence_2 = \"I love studying statistics\"\n",
    "sentence_3 = \"I love working as Data Scientist\"\n",
    "\n",
    "# so, lets respresent this in a numerical way\n",
    "# We'll at first combine all these sentences and then tokenize and get distinct tokens.\n",
    "# we basically combine the sentences with spaces between them.\n",
    "text = \" \".join([sentence_1, sentence_2, sentence_3])\n",
    "\n",
    "# now we tokenize and get the distinct tokens as a list by using set() function.\n",
    "tokens = list(set(nltk.word_tokenize(text)))\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The length of the numerical representation of sentences (numerical vector) will be the length of this list. Since we have six distinct words, we'll represent each sentence with 6 digits. To numerically represent each sentence, let's assign 1 to each occurence of words above. So, if a word form th tokenized list is inside the sentence, it's 1; if it's not, it's 0. Beware that we are not dealing with the order of words inside the sentence.\n",
    "\n",
    "['working', 'Data', 'statistics', 'as', 'studying', 'I', 'AI', 'love', 'Scientist']\n",
    "\n",
    "By looking at the reference list, we can represent sentence_1 (\"I love studying AI\") as below:  \n",
    "So, the numerical representation of sentence_1 is: `[0, 0, 0, 0, 1, 1, 1, 1, 0]`\n",
    "\n",
    "Simply said, a bag-of-words is a representation of text that describes the occurrence of words within a document. That is, each document or data entry is represented as numerical vectors based on the vocabulary built from the corpora. The intuition is that documents are similar if they have similar content. \n",
    "\n",
    "It is called a “bag” of words, because any information about the order or structure of words in the document is discarded. The model is only concerned with whether known words occur in the document, not where in the document.\n",
    "\n",
    "As you can see, while working with large texts, there will be multiple occurences for some words. And we need to repreesent this as well. Here comes term frequency. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Count Vectorizer \n",
    "\n",
    "With count vectorizer, we count the appearance of the words in each text. Using the fit method, our Count Vectorizer will “learn” what tokens are being used in our sentences inside the corpus. That is, we at first train our Count Vectorizer model with the training set and then let the model learn the tokens. Then, whenever we throw a new sentence to this model, our Count Vectorizer model will show use the matrix representation of that sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vect = CountVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "ss = \"bear by thornton w burgess 1920\\r\\n\\r\\ni\\r\\n\\r\\nbuster bear goes fishing\\r\\n\\r\\n\\r\\nbuster\\n\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'bear by thornton w burgess 1920 i buster bear goes fishing buster\\n'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pattern = re.compile('(\\r\\n)+')\n",
    "pattern2 = re.compile('\\n+')\n",
    "re.sub(pattern, ' ', ss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'bear by thornton w burgess 1920 i buster bear goes fishing buster'"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ss = re.sub(pattern, ' ', ss)\n",
    "re.sub(pattern2, '', ss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['adventures buster bear thornton w. burgess 1920 buster bear goes fishing buster bear yawned lay comfortable bed leaves watched first early morning sunbeams creeping green forest chase black shadows',\n",
       " 'yawned slowly got feet shook',\n",
       " 'walked big pine-tree stood hind legs reached high trunk tree could scratched bark great claws',\n",
       " 'yawned seemed jaws would crack sat think wanted breakfast',\n",
       " 'sat trying make mind would taste best listening sounds told waking little people live green forest']"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pattern = re.compile('(\\r\\n)+')  # Windows new line \n",
    "clean_text = re.sub(pattern, ' ', burgess)\n",
    "clean_sent = nltk.sent_tokenize(clean_text)\n",
    "\n",
    "# Now remove the punctuation and stopwords from each sentence\n",
    "clean_corpus = []\n",
    "for sent in clean_sent:\n",
    "    words = nltk.word_tokenize(sent)\n",
    "    remove_punc = [word for word in words if word not in string.punctuation]\n",
    "    clean_corpus.append(' '.join([word for word in remove_punc if not word in stop_words]))\n",
    "\n",
    "clean_corpus[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=None, vocabulary=None)"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vect.fit(clean_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['112', '1920', '26', '45', '_71_', '_could_', '_had_', '_has_', '_him_', '_his_', '_i', '_our_', '_page', '_their_', '_was_', 'ability', 'able', 'accept', 'according', 'acquainted', 'across', 'actually', 'admit', 'admitted', 'adventures', 'advice', 'advised', 'afraid', 'afterward', 'age', 'ago', 'agreed', 'ah', 'air', 'almost', 'alone', 'along', 'already', 'also', 'although', 'altogether', 'always', 'among', 'amounted', 'anger', 'angrier', 'angrily', 'angry', 'another', 'anybody']\n"
     ]
    }
   ],
   "source": [
    "print(vect.get_feature_names()[:50])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = vect.transform(clean_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>112</th>\n",
       "      <th>1920</th>\n",
       "      <th>26</th>\n",
       "      <th>45</th>\n",
       "      <th>_71_</th>\n",
       "      <th>_could_</th>\n",
       "      <th>_had_</th>\n",
       "      <th>_has_</th>\n",
       "      <th>_him_</th>\n",
       "      <th>_his_</th>\n",
       "      <th>...</th>\n",
       "      <th>year</th>\n",
       "      <th>years</th>\n",
       "      <th>yell</th>\n",
       "      <th>yelled</th>\n",
       "      <th>yelling</th>\n",
       "      <th>yes</th>\n",
       "      <th>yesterday</th>\n",
       "      <th>yet</th>\n",
       "      <th>yo</th>\n",
       "      <th>young</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1397 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   112  1920  26  45  _71_  _could_  _had_  _has_  _him_  _his_  ...    year  \\\n",
       "0    0     1   0   0     0        0      0      0      0      0  ...       0   \n",
       "1    0     0   0   0     0        0      0      0      0      0  ...       0   \n",
       "2    0     0   0   0     0        0      0      0      0      0  ...       0   \n",
       "3    0     0   0   0     0        0      0      0      0      0  ...       0   \n",
       "4    0     0   0   0     0        0      0      0      0      0  ...       0   \n",
       "\n",
       "   years  yell  yelled  yelling  yes  yesterday  yet  yo  young  \n",
       "0      0     0       0        0    0          0    0   0      0  \n",
       "1      0     0       0        0    0          0    0   0      0  \n",
       "2      0     0       0        0    0          0    0   0      0  \n",
       "3      0     0       0        0    0          0    0   0      0  \n",
       "4      0     0       0        0    0          0    0   0      0  \n",
       "\n",
       "[5 rows x 1397 columns]"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's convert this to pandas dataframe(a dense matrix version) for better intuition.\n",
    "import pandas as pd\n",
    "cv_df = pd.DataFrame(matrix.toarray(), columns=vect.get_feature_names())\n",
    "cv_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we now have all the frequencies of each token inside a sentence. \n",
    "\n",
    "We have 1397 different features (number of columns in dataframe) created from all of our sentences. This means each row will mostly be filled with zeros. In order to save space/computational power a sparse matrix is created. This means that only the location and value of non-zero values is saved. Since we're transforming the same sentences into a model fitted from those sentences, this is not an issue at the moment. But when we try to find a matrix representation of another sentence other than we already have, it's quite possible that many of thee features (tokens) will not be in the next sentence and all column would be zero. So the idea behind the sparse matrix is to save some space because of that issue.\n",
    "\n",
    "However, if we'd like to feed a new sentence into this model and try to get matrix representation, we may have an issue though. Since our model is fitted on the sentences we already have, if a new sentence doesn't have a token that could be recognized by our model, that token wouldn't be represented. In this situation we simply neeed to include append our new message to our original collection and then refit and transform to make sure we don’t lose this information. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TfidfVectorizer (Term Frequency – Inverse Document Frequency)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A problem with scoring word frequency is that highly frequent words start to dominate in the document (larger score), but may not contain as much “informational content” to the model as rarer but perhaps domain specific words. One approach is to rescale the frequency of words by how often they appear in all documents, so that the scores for frequent words like “the” that are also frequent across all documents are penalized. This approach to scoring is called Term Frequency – Inverse Document Frequency, or TF-IDF for short, where:\n",
    "\n",
    "Term Frequency: is a scoring of the frequency of the word in the current document.\n",
    "Inverse Document Frequency: is a scoring of how rare the word is across documents.\n",
    "\n",
    "The scores are a weighting where not all words are equally as important or interesting. The scores have the effect of highlighting words that are distinct (contain useful information) in a given document. Just like CountVectorizer, TfidfVectorizer also creates a document term matrix (DTM) from our messages. However, instead of filling the DTM with token counts it calculates term frequency-inverse document frequency value for each word(TF-IDF). The TF-IDF is the product of two weights, the term frequency and the inverse document frequency.\n",
    "\n",
    "To generalize: TF-IDF = term frequency * (1 / document frequency)\n",
    "\n",
    "Or: TF-IDF = term frequency * inverse document frequency\n",
    "\n",
    "Term frequency is a weight representing how often a word occurs in a document. If we have several occurences of the same word in one document we can expect the TF-IDF to increase. Inverse document frequency is another weight representing how common a word is across documents. If a word is used in many documents then the TF-IDF will decrease. With the definition out of the way we’ll go through a few examples to see how it works. Since the usage is pretty much identical to CountVectorizer and we’ll be going through a few examples we’ll make a function to create a DTM from our messages to make things a bit easier and clearer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['112', '1920', '26', '45', '_71_', '_could_', '_had_', '_has_', '_him_', '_his_', '_i', '_our_', '_page', '_their_', '_was_', 'ability', 'able', 'accept', 'according', 'acquainted', 'across', 'actually', 'admit', 'admitted', 'adventures', 'advice', 'advised', 'afraid', 'afterward', 'age', 'ago', 'agreed', 'ah', 'air', 'almost', 'alone', 'along', 'already', 'also', 'although', 'altogether', 'always', 'among', 'amounted', 'anger', 'angrier', 'angrily', 'angry', 'another', 'anybody']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "vect = TfidfVectorizer()\n",
    "vect.fit(clean_corpus)\n",
    "print(vect.get_feature_names()[:50])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>112</th>\n",
       "      <th>1920</th>\n",
       "      <th>26</th>\n",
       "      <th>45</th>\n",
       "      <th>_71_</th>\n",
       "      <th>_could_</th>\n",
       "      <th>_had_</th>\n",
       "      <th>_has_</th>\n",
       "      <th>_him_</th>\n",
       "      <th>_his_</th>\n",
       "      <th>...</th>\n",
       "      <th>year</th>\n",
       "      <th>years</th>\n",
       "      <th>yell</th>\n",
       "      <th>yelled</th>\n",
       "      <th>yelling</th>\n",
       "      <th>yes</th>\n",
       "      <th>yesterday</th>\n",
       "      <th>yet</th>\n",
       "      <th>yo</th>\n",
       "      <th>young</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.228444</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1397 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   112      1920   26   45  _71_  _could_  _had_  _has_  _him_  _his_  ...    \\\n",
       "0  0.0  0.228444  0.0  0.0   0.0      0.0    0.0    0.0    0.0    0.0  ...     \n",
       "1  0.0  0.000000  0.0  0.0   0.0      0.0    0.0    0.0    0.0    0.0  ...     \n",
       "2  0.0  0.000000  0.0  0.0   0.0      0.0    0.0    0.0    0.0    0.0  ...     \n",
       "3  0.0  0.000000  0.0  0.0   0.0      0.0    0.0    0.0    0.0    0.0  ...     \n",
       "4  0.0  0.000000  0.0  0.0   0.0      0.0    0.0    0.0    0.0    0.0  ...     \n",
       "\n",
       "   year  years  yell  yelled  yelling  yes  yesterday  yet   yo  young  \n",
       "0   0.0    0.0   0.0     0.0      0.0  0.0        0.0  0.0  0.0    0.0  \n",
       "1   0.0    0.0   0.0     0.0      0.0  0.0        0.0  0.0  0.0    0.0  \n",
       "2   0.0    0.0   0.0     0.0      0.0  0.0        0.0  0.0  0.0    0.0  \n",
       "3   0.0    0.0   0.0     0.0      0.0  0.0        0.0  0.0  0.0    0.0  \n",
       "4   0.0    0.0   0.0     0.0      0.0  0.0        0.0  0.0  0.0    0.0  \n",
       "\n",
       "[5 rows x 1397 columns]"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix = vect.transform(clean_corpus)\n",
    "tf_df = pd.DataFrame(matrix.toarray(), columns=vect.get_feature_names())\n",
    "tf_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bear           0.279688\n",
       "buster         0.244090\n",
       "thornton       0.228444\n",
       "1920           0.228444\n",
       "shadows        0.228444\n",
       "chase          0.228444\n",
       "creeping       0.228444\n",
       "burgess        0.228444\n",
       "comfortable    0.228444\n",
       "black          0.215604\n",
       "Name: 0, dtype: float64"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now look at the first sentence by sorting the tfidf scores in a decreasing order\n",
    "tf_df.iloc[0].sort_values(ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's apply these NLP procedures with Spacy package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applying all these processes with Spacy package is quite easy and straightforward. We should decide which package to use by checking the accuracy or any other metrics that we want to deal with. We at first import Spacy package and load \"English\" parser. Then we dump any text into this parser and Spacy will apply various preprocssing mthods as default. All we need to do is to usee relevant functions to extract what we wamt such as tokens, lemmas, pos etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "galaxies\n",
      "recede\n",
      "from\n",
      "us\n",
      "at\n",
      "speeds\n",
      "proportional\n",
      "to\n",
      "their\n",
      "distances\n",
      ",\n",
      "going\n",
      "faster\n",
      "the\n",
      "farther\n",
      "away\n",
      "they\n",
      "are\n",
      ".\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "en = spacy.load('en')\n",
    "doc = en('galaxies recede from us at speeds proportional to their distances, going faster the farther away they are.')\n",
    "\n",
    "# now text is parsed by nearly all the basic NLP techniques and we are ready to extract what we need.\n",
    "# tokenization \n",
    "for token in doc:\n",
    "    print(token.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lemmatization and POS**\n",
    "\n",
    "- **text**: The original word text.\n",
    "- **lemma**: The base form of the word.\n",
    "- **pos**: The simple part-of-speech tag.\n",
    "- **tag**: The detailed part-of-speech tag.\n",
    "- **dep**: Syntactic dependency, i.e. the relation between tokens.\n",
    "- **shape**: The word shape – capitalisation, punctuation, digits.\n",
    "- **is_alpha**: Is the token an alpha character?\n",
    "- **is_stop**: Is the token part of a stop list, i.e. the most common words of the language?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "galaxies==>galaxy==>NOUN==>NNS==>nsubj==>xxxx==>True==>False\n",
      "recede==>recede==>VERB==>VBP==>ROOT==>xxxx==>True==>False\n",
      "from==>from==>ADP==>IN==>prep==>xxxx==>True==>True\n",
      "us==>-PRON-==>PRON==>PRP==>pobj==>xx==>True==>True\n",
      "at==>at==>ADP==>IN==>prep==>xx==>True==>True\n",
      "speeds==>speed==>NOUN==>NNS==>compound==>xxxx==>True==>False\n",
      "proportional==>proportional==>ADJ==>JJ==>pobj==>xxxx==>True==>False\n",
      "to==>to==>ADP==>IN==>prep==>xx==>True==>True\n",
      "their==>-PRON-==>ADJ==>PRP$==>poss==>xxxx==>True==>True\n",
      "distances==>distance==>NOUN==>NNS==>pobj==>xxxx==>True==>False\n",
      ",==>,==>PUNCT==>,==>punct==>,==>False==>False\n",
      "going==>go==>VERB==>VBG==>advcl==>xxxx==>True==>False\n",
      "faster==>fast==>ADV==>RBR==>advmod==>xxxx==>True==>False\n",
      "the==>the==>DET==>DT==>det==>xxx==>True==>True\n",
      "farther==>far==>ADV==>RB==>advmod==>xxxx==>True==>False\n",
      "away==>away==>ADV==>RB==>advmod==>xxxx==>True==>False\n",
      "they==>-PRON-==>PRON==>PRP==>nsubj==>xxxx==>True==>True\n",
      "are==>be==>VERB==>VBP==>ccomp==>xxx==>True==>True\n",
      ".==>.==>PUNCT==>.==>punct==>.==>False==>False\n"
     ]
    }
   ],
   "source": [
    "for token in doc:\n",
    "    print(token.text, token.lemma_, token.pos_, token.tag_,\n",
    "         token.dep_, token.shape_, token.is_alpha, token.is_stop, sep=\" ==> \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tokens</th>\n",
       "      <th>lemmas</th>\n",
       "      <th>pos</th>\n",
       "      <th>tag</th>\n",
       "      <th>dep</th>\n",
       "      <th>shape</th>\n",
       "      <th>is_alpha</th>\n",
       "      <th>is_stop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>galaxies</td>\n",
       "      <td>galaxy</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NNS</td>\n",
       "      <td>nsubj</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>recede</td>\n",
       "      <td>recede</td>\n",
       "      <td>VERB</td>\n",
       "      <td>VBP</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>from</td>\n",
       "      <td>from</td>\n",
       "      <td>ADP</td>\n",
       "      <td>IN</td>\n",
       "      <td>prep</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>us</td>\n",
       "      <td>-PRON-</td>\n",
       "      <td>PRON</td>\n",
       "      <td>PRP</td>\n",
       "      <td>pobj</td>\n",
       "      <td>xx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>at</td>\n",
       "      <td>at</td>\n",
       "      <td>ADP</td>\n",
       "      <td>IN</td>\n",
       "      <td>prep</td>\n",
       "      <td>xx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>speeds</td>\n",
       "      <td>speed</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NNS</td>\n",
       "      <td>compound</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>proportional</td>\n",
       "      <td>proportional</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>JJ</td>\n",
       "      <td>pobj</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>to</td>\n",
       "      <td>to</td>\n",
       "      <td>ADP</td>\n",
       "      <td>IN</td>\n",
       "      <td>prep</td>\n",
       "      <td>xx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>their</td>\n",
       "      <td>-PRON-</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>PRP$</td>\n",
       "      <td>poss</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>distances</td>\n",
       "      <td>distance</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NNS</td>\n",
       "      <td>pobj</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>,</td>\n",
       "      <td>,</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>,</td>\n",
       "      <td>punct</td>\n",
       "      <td>,</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>going</td>\n",
       "      <td>go</td>\n",
       "      <td>VERB</td>\n",
       "      <td>VBG</td>\n",
       "      <td>advcl</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>faster</td>\n",
       "      <td>fast</td>\n",
       "      <td>ADV</td>\n",
       "      <td>RBR</td>\n",
       "      <td>advmod</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>the</td>\n",
       "      <td>the</td>\n",
       "      <td>DET</td>\n",
       "      <td>DT</td>\n",
       "      <td>det</td>\n",
       "      <td>xxx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>farther</td>\n",
       "      <td>far</td>\n",
       "      <td>ADV</td>\n",
       "      <td>RB</td>\n",
       "      <td>advmod</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>away</td>\n",
       "      <td>away</td>\n",
       "      <td>ADV</td>\n",
       "      <td>RB</td>\n",
       "      <td>advmod</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>they</td>\n",
       "      <td>-PRON-</td>\n",
       "      <td>PRON</td>\n",
       "      <td>PRP</td>\n",
       "      <td>nsubj</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>are</td>\n",
       "      <td>be</td>\n",
       "      <td>VERB</td>\n",
       "      <td>VBP</td>\n",
       "      <td>ccomp</td>\n",
       "      <td>xxx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>.</td>\n",
       "      <td>.</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>.</td>\n",
       "      <td>punct</td>\n",
       "      <td>.</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          tokens        lemmas    pos   tag       dep shape  is_alpha  is_stop\n",
       "0       galaxies        galaxy   NOUN   NNS     nsubj  xxxx      True    False\n",
       "1         recede        recede   VERB   VBP      ROOT  xxxx      True    False\n",
       "2           from          from    ADP    IN      prep  xxxx      True     True\n",
       "3             us        -PRON-   PRON   PRP      pobj    xx      True     True\n",
       "4             at            at    ADP    IN      prep    xx      True     True\n",
       "5         speeds         speed   NOUN   NNS  compound  xxxx      True    False\n",
       "6   proportional  proportional    ADJ    JJ      pobj  xxxx      True    False\n",
       "7             to            to    ADP    IN      prep    xx      True     True\n",
       "8          their        -PRON-    ADJ  PRP$      poss  xxxx      True     True\n",
       "9      distances      distance   NOUN   NNS      pobj  xxxx      True    False\n",
       "10             ,             ,  PUNCT     ,     punct     ,     False    False\n",
       "11         going            go   VERB   VBG     advcl  xxxx      True    False\n",
       "12        faster          fast    ADV   RBR    advmod  xxxx      True    False\n",
       "13           the           the    DET    DT       det   xxx      True     True\n",
       "14       farther           far    ADV    RB    advmod  xxxx      True    False\n",
       "15          away          away    ADV    RB    advmod  xxxx      True    False\n",
       "16          they        -PRON-   PRON   PRP     nsubj  xxxx      True     True\n",
       "17           are            be   VERB   VBP     ccomp   xxx      True     True\n",
       "18             .             .  PUNCT     .     punct     .     False    False"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#lets view this parsing in Pandas dataframe\n",
    "\n",
    "tokens = [token.text for token in doc]\n",
    "lemmas = [token.lemma_ for token in doc]\n",
    "pos = [token.pos_ for token in doc]\n",
    "tag = [token.tag_ for token in doc]\n",
    "dep = [token.dep_ for token in doc]\n",
    "shape = [token.shape_ for token in doc]\n",
    "is_alpha = [token.is_alpha for token in doc]\n",
    "is_stop = [token.is_stop for token in doc]\n",
    "\n",
    "pd.DataFrame({\"tokens\":tokens, \"lemmas\":lemmas, \"pos\":pos, \"tag\":tag, \n",
    "             \"dep\": dep, \"shape\":shape, \"is_alpha\":is_alpha, \"is_stop\":is_stop}, \n",
    "             columns=[\"tokens\", \"lemmas\", \"pos\", \"tag\", \"dep\", \"shape\", \"is_alpha\", \"is_stop\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **NER**: Named entities are available as the ***ents*** property of a doc:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Steve Jobs PERSON\n",
      "Apple Inc. ORG\n",
      "San Francisco GPE\n"
     ]
    }
   ],
   "source": [
    "en = spacy.load('en')\n",
    "doc = en(\"Steve Jobs, the CEO of Apple Inc. is living in San Francisco.\")\n",
    "\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that, without applying any other procedure, with just a few lines of code, Steve and Jobs are recognized as a whole person."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Topic Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Topic modeling is a process of automatically identifying the topics present in a text corpus, it derives the hidden patterns among the words in the corpus in an unsupervised manner. Topics are defined as “a repeating pattern of co-occurring terms in a corpus”. Topic modelling can be described as a method for finding a group of words (i.e topic) from a collection of documents that best represents the information in the collection.\n",
    "\n",
    "As the name suggests, it is a process to automatically identify topics present in a text object and to derive hidden patterns exhibited by a text corpus. Thus, assisting better decision making. \n",
    "\n",
    "Topic Modelling is different from rule-based text mining approaches that use regular expressions or dictionary based keyword searching techniques. It is an unsupervised approach used for finding and observing the bunch of words (called “topics”) in large clusters of texts.\n",
    "\n",
    "A good topic model results in – “health”, “doctor”, “patient”, “hospital” for a topic – Healthcare, and “farm”, “crops”, “wheat” for a topic – “Farming”.\n",
    "\n",
    "Topic Models are very useful for the purpose for document clustering, organizing large blocks of textual data, information retrieval from unstructured text and feature selection. For Example – New York Times are using topic models to boost their user – article recommendation engines. Various professionals are using topic models for recruitment industries where they aim to extract latent features of job descriptions and map them to right candidates. They are being used to organize large datasets of emails, customer reviews, and user social media profiles.\n",
    "\n",
    "There are many approaches for obtaining topics from a text such as – Term Frequency and Inverse Document Frequency (TfIdf). NonNegative Matrix Factorization techniques. Latent Dirichlet Allocation(LDA) is the most popular topic modeling technique.\n",
    "\n",
    "LDA assumes documents are produced from a mixture of topics. Those topics then generate words based on their probability distribution. Given a dataset of documents, LDA backtracks and tries to figure out what topics would create those documents in the first place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets see how it works with the following sentences.\n",
    "\n",
    "doc1 = \"I have big exam tomorrow and I need to study hard to get a good grade.\"\n",
    "doc2 = \"My wife likes to go out with me but I prefer staying at home and studying.\"\n",
    "doc3 = \"Kids are playing football in the field and they seem to have fun\"\n",
    "doc4 = \"Sometimes I feel depressed while driving and it's hard to focus on the road.\"\n",
    "doc5 = \"I usually prefer reading at home but my wife prefers watching a TV.\"\n",
    "\n",
    "# array of documents aka corpus\n",
    "corpus = [doc1, doc2, doc3, doc4, doc5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['big', 'exam', 'tomorrow', 'need', 'study', 'hard', 'get', 'good', 'grade'],\n",
       " ['wife', 'likes', 'go', 'prefer', 'staying', 'home', 'studying'],\n",
       " ['kids', 'playing', 'football', 'field', 'seem', 'fun'],\n",
       " ['sometimes', 'feel', 'depressed', 'driving', 'hard', 'focus', 'road'],\n",
       " ['usually', 'prefer', 'reading', 'home', 'wife', 'prefers', 'watching', 'tv']]"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = nltk.tokenize.RegexpTokenizer(r'\\w+')\n",
    "token_list = [tokenizer.tokenize(sentence.lower()) for sentence in corpus]\n",
    "\n",
    "def remove_stopwords(words):\n",
    "    return [word for word in words if word not in stop_words]\n",
    "\n",
    "tokenized_data = [remove_stopwords(token) for token in token_list]\n",
    "tokenized_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The LDA model discovers the different topics that the documents represent and how much of each topic is present in a document. \n",
    "\n",
    "Python provides many great libraries for text mining practices, “gensim” is one such clean and beautiful library to handle text data. It is scalable, robust and efficient. Following code shows how to convert a corpus into a document-term matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic #0: 0.054*\"hard\" + 0.054*\"wife\" + 0.054*\"tv\" + 0.054*\"grade\" + 0.054*\"watching\"\n",
      "Topic #1: 0.118*\"kids\" + 0.118*\"playing\" + 0.118*\"football\" + 0.118*\"field\" + 0.118*\"seem\"\n",
      "Topic #2: 0.107*\"sometimes\" + 0.107*\"focus\" + 0.107*\"road\" + 0.107*\"driving\" + 0.107*\"feel\"\n",
      "Topic #3: 0.030*\"home\" + 0.030*\"wife\" + 0.030*\"prefer\" + 0.030*\"hard\" + 0.030*\"field\"\n",
      "Topic #4: 0.030*\"prefer\" + 0.030*\"home\" + 0.030*\"hard\" + 0.030*\"wife\" + 0.030*\"playing\"\n",
      "Topic #5: 0.103*\"studying\" + 0.101*\"likes\" + 0.101*\"go\" + 0.096*\"home\" + 0.094*\"wife\"\n",
      "Topic #6: 0.030*\"prefer\" + 0.030*\"home\" + 0.030*\"hard\" + 0.030*\"prefers\" + 0.030*\"seem\"\n",
      "Topic #7: 0.092*\"prefer\" + 0.085*\"staying\" + 0.080*\"wife\" + 0.078*\"home\" + 0.070*\"go\"\n",
      "Topic #8: 0.030*\"home\" + 0.030*\"prefer\" + 0.030*\"wife\" + 0.030*\"football\" + 0.030*\"field\"\n",
      "Topic #9: 0.030*\"hard\" + 0.030*\"prefer\" + 0.030*\"home\" + 0.030*\"wife\" + 0.030*\"staying\"\n"
     ]
    }
   ],
   "source": [
    "from gensim import corpora, models\n",
    "\n",
    "# Build a Dictionary - association word to numeric id\n",
    "dictionary = corpora.Dictionary(tokenized_data)\n",
    " \n",
    "# Transform the collection of texts to a numerical form\n",
    "corpus = [dictionary.doc2bow(text) for text in tokenized_data]\n",
    "\n",
    "# We are asking LDA to find 10 topics in the data\n",
    "lda_model = models.LdaModel(corpus=corpus, num_topics=10, id2word=dictionary)\n",
    "\n",
    "for idx in range(10):\n",
    "    # Print the first 10 most representative topics\n",
    "    print(\"Topic #%s:\" % idx, lda_model.print_topic(idx, 5))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we trained and built our LDA model over the five simple sentences, whenever we want to detect the topic of a new sentence or text, we'll at first prepare the text and then push that into our model to get a topic. Let's try to predict a topic for a new sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 0.033337917),\n",
       " (1, 0.03333333),\n",
       " (2, 0.03333333),\n",
       " (3, 0.033333335),\n",
       " (4, 0.033333335),\n",
       " (5, 0.6999899),\n",
       " (6, 0.033333335),\n",
       " (7, 0.033338852),\n",
       " (8, 0.033333335),\n",
       " (9, 0.033333335)]"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_sentence = \"My wife plans to go out tonight\"\n",
    "lda_model.get_document_topics(dictionary.doc2bow(new_sentence.split()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as you can see, topic-5 (listed above) is the most relevant topic for this sentence. \n",
    "\n",
    "Topic #5: 0.103*\"studying\" + 0.101*\"likes\" + 0.101*\"go\" + 0.096*\"home\" + 0.094*\"wife\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}