Notebooks/02_Pre_Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2f17c74a",
   "metadata": {
    "papermill": {
     "duration": 0.007302,
     "end_time": "2023-02-20T20:23:52.207360",
     "exception": false,
     "start_time": "2023-02-20T20:23:52.200058",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<center>\n",
    "<img src='../assets/1/lang-pic.jpg' width=600>\n",
    "</center>\n",
    "    \n",
    "# 1. Introduction\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">1.1 NLP series</p>\n",
    "\n",
    "This is the **second in a series of notebooks** covering the **fundamentals of Natural Language Processing (NLP)**. I find that the best way to learn is by teaching others, hence why I am sharing my journey learning this field from scratch. I hope these notebooks can be helpful to you too. \n",
    "\n",
    "NLP series:\n",
    "\n",
    "1. [Tokenization](./01_Tokenization.ipynb)\n",
    "2. Preprocessing \n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/JUSTSUJAY/NLP_One_Shot/blob/28eb64d1c9db75ec790b0945aee7f533f0c52ecd/Notebooks/02_Pre_Processing.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/JUSTSUJAY/NLP_One_Shot/blob/main/Notebooks/02_Pre_Processing.ipynb)\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">1.2 Outline</p>\n",
    "\n",
    "Last time, we saw how to load a language model and tokenize a string of text. This notebook focuses on **further pre-processing steps** we can perform on tokens. In particular, we will begin by looking at **case folding**, **stop word removal**, **stemming** and **lemmatization**. \n",
    "\n",
    "We will then examine some more **advanced** pre-processing techniques, namely **part-of-speech tagging** and **named entity recognition**, which are useful tasks in of themselves. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7f944dc",
   "metadata": {
    "papermill": {
     "duration": 0.006188,
     "end_time": "2023-02-20T20:23:52.219969",
     "exception": false,
     "start_time": "2023-02-20T20:23:52.213781",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2. Basic pre-processing\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.1 Case folding</p>\n",
    "\n",
    "This is the act of converting every token to be uniformly **lower case** or **upper case**. \n",
    "\n",
    "<center>\n",
    "<img src='../assets/2/case-folding.jpg' width=600>\n",
    "</center>\n",
    "    \n",
    "This can be beneficial because it will **reduce the number of unique tokens** in a corpus, i,e. the size of the **vocabulary**, hence make the processing of these tokens more memory and computational effecient. The downside however is **information loss**. \n",
    "\n",
    "For example `\"Green\"` (name) has a different meaning to `\"green\"` (colour) but both would get the **same token** if case folding is applied. Whether it makes sense to use case folding **depends on the application** (is speed or accuracy more important)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "82f96667",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:23:52.235267Z",
     "iopub.status.busy": "2023-02-20T20:23:52.234529Z",
     "iopub.status.idle": "2023-02-20T20:24:03.654973Z",
     "shell.execute_reply": "2023-02-20T20:24:03.653920Z"
    },
    "papermill": {
     "duration": 11.431265,
     "end_time": "2023-02-20T20:24:03.657666",
     "exception": false,
     "start_time": "2023-02-20T20:23:52.226401",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spacy 3.3.1\n"
     ]
    }
   ],
   "source": [
    "# Import spacy library\n",
    "import spacy\n",
    "print(spacy.__name__, spacy.__version__)\n",
    "\n",
    "# Load language model\n",
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17135708",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-17T22:18:06.566835Z",
     "iopub.status.busy": "2022-12-17T22:18:06.566080Z",
     "iopub.status.idle": "2022-12-17T22:18:06.575330Z",
     "shell.execute_reply": "2022-12-17T22:18:06.573512Z",
     "shell.execute_reply.started": "2022-12-17T22:18:06.566797Z"
    },
    "papermill": {
     "duration": 0.006497,
     "end_time": "2023-02-20T20:24:03.671121",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.664624",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To case fold to lower cases we can use the `.lower` attribute. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "91580ea7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:03.687199Z",
     "iopub.status.busy": "2023-02-20T20:24:03.685926Z",
     "iopub.status.idle": "2023-02-20T20:24:03.711978Z",
     "shell.execute_reply": "2023-02-20T20:24:03.710571Z"
    },
    "papermill": {
     "duration": 0.036708,
     "end_time": "2023-02-20T20:24:03.714580",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.677872",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the', 'train', 'to', 'london', 'leaves', 'at', '10', 'am', 'on', 'tuesday', '.']\n"
     ]
    }
   ],
   "source": [
    "# Tokenize\n",
    "s = \"The train to London leaves at 10am on Tuesday.\"\n",
    "doc = nlp(s)\n",
    "\n",
    "# Case fold\n",
    "print([t.lower_ for t in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c54c24c",
   "metadata": {
    "papermill": {
     "duration": 0.006483,
     "end_time": "2023-02-20T20:24:03.729313",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.722830",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We might want to be **more granular** and only case fold if certain conditions are met. For example, we could **skip the first word** in a sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "11c37cc6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:03.743958Z",
     "iopub.status.busy": "2023-02-20T20:24:03.743622Z",
     "iopub.status.idle": "2023-02-20T20:24:03.749749Z",
     "shell.execute_reply": "2023-02-20T20:24:03.748271Z"
    },
    "papermill": {
     "duration": 0.015818,
     "end_time": "2023-02-20T20:24:03.751828",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.736010",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['The', 'train', 'to', 'london', 'leaves', 'at', '10', 'am', 'on', 'tuesday', '.']\n"
     ]
    }
   ],
   "source": [
    "# Conditional case folding\n",
    "print([t.lower_ if not t.is_sent_start else t.text for t in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8050665e",
   "metadata": {
    "papermill": {
     "duration": 0.007282,
     "end_time": "2023-02-20T20:24:03.766059",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.758777",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.2 Stop word removal</p>\n",
    "\n",
    "Stop words are words that **appear commonly** but **carry little information**. Examples include, `\"a\"`, `\"the\"`, `\"of\"`, `\"an\"`, `\"this\"`,`\"that\"`. Similar to case folding, removing stop words can **improve efficiency** but comes at the cost of **losing contextual information**. \n",
    "\n",
    "<center>\n",
    "<img src='../assets/2/stop-word-removal.jpg' width=600>\n",
    "</center>\n",
    "\n",
    "The choice of whether to use stop word removal will depend on the task being performed. For some tasks like **topic modelling** (identifying topics in text), contextual information is not as **important** compared to a task like **sentiment analysis** where the stop word `\"not\"` can change the sentiment completely. \n",
    "\n",
    "Also note that different libraries have **different** stop word lists so you might want to **tune** your list depending on the application. Spacy's language model has **over 300 stop words**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5f566e90",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:03.781503Z",
     "iopub.status.busy": "2023-02-20T20:24:03.781080Z",
     "iopub.status.idle": "2023-02-20T20:24:03.787318Z",
     "shell.execute_reply": "2023-02-20T20:24:03.786205Z"
    },
    "papermill": {
     "duration": 0.016859,
     "end_time": "2023-02-20T20:24:03.789928",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.773069",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'seems', 'the', 'whatever', 'front', 'hereby', 'nevertheless', 'else', 'somewhere', 'toward', 'keep', 'top', 'get', 'across', 'latter', 'n‘t', '’ve', 'perhaps', 'everyone', 'will', 'somehow', 'even', 'whole', 'together', \"'m\", 'twenty', 'thereby', 'made', 'each', 'what', 'have', 'of', 'just', 'really', 'must', 'call', 'bottom', 'hers', 'never', 'wherever', 'latterly', 'why', '’re', 'part', 'becomes', 'might', 'along', 'am', 'others', 'formerly', 'fifty', 'thus', 'namely', 'one', 'an', 'towards', 'many', 'upon', 'them', 'my', 'always', 'several', 'unless', 'fifteen', 'whoever', 'herself', 'quite', '‘re', 'nowhere', '’m', 'until', 'whether', 'three', 'via', 'something', 'may', 'ever', 'mine', 'herein', 'more', 'least', 'be', 'own', 'nine', \"'d\", 'therein', 'already', 'whereafter', 'amount', 'because', 'has', 'every', 'now', 'then', 'regarding', 'sometime', 're', 'side', 'per', 'alone', 'all', 'once', 'out', 'someone', 'thence', 'next', 'through', 'me', 'yours', 'four', 'itself', 'used', 'not', 'yourselves', 'against', 'whence', 'first', 'up', 'eight', 'while', 'below', 'over', 'here', 'throughout', 'put', 'much', \"'re\", 'sometimes', 'onto', 'been', 'also', 'but', 'though', 'n’t', 'which', 'hereupon', 'down', 'last', \"'ll\", 'those', 'ours', 'empty', 'him', 'except', 'cannot', 'behind', 'beyond', 'from', 'afterwards', 'anyhow', 'around', 'thereupon', 'anywhere', 'indeed', 'where', 'however', 'so', 'thereafter', 'two', 'as', 'myself', 'sixty', 'often', 'moreover', 'above', 'on', 'would', '‘d', 'how', 'neither', 'third', 'wherein', 'seem', 'there', 'any', 'for', 'should', 'some', 'himself', 'otherwise', '‘ll', '’s', 'since', 'could', 'show', 'twelve', 'anything', 'does', 'their', 'back', 'forty', 'yet', 'make', 'than', 'you', 'same', 'yourself', 'whereupon', 'move', 'had', 'doing', 'further', 'see', 'become', 'everything', 'again', 'between', 'did', 'became', 'elsewhere', 'she', 'us', 'can', 'whom', 'go', 'is', 'please', 'without', 'another', 'full', 'anyway', 'that', 'after', 'becoming', 'serious', 'whereas', 'and', 'ourselves', 'other', 'most', 'ca', \"n't\", 'its', 'using', 'with', 'everywhere', 'thru', 'due', '‘s', 'such', 'among', 'less', 'by', 'therefore', '‘ve', 'do', '’d', 'before', 'whose', 'whereby', 'seemed', 'mostly', 'it', 'into', 'six', 'take', 'amongst', 'ten', 'during', '’ll', 'nothing', 'almost', 'although', 'eleven', 'still', 'nor', 'noone', 'name', 'this', 'were', 'done', 'only', 'he', 'a', 'in', 'both', 'enough', 'five', 'few', 'about', 'beside', '‘m', 'hence', 'meanwhile', 'say', 'none', 'his', 'too', 'either', 'our', 'whenever', 'or', 'give', 'these', 'at', 'nobody', 'to', 'various', 'whither', \"'ve\", 'her', 'hundred', 'seeming', 'besides', 'are', 'we', 'former', 'when', 'hereafter', 'i', 'themselves', 'they', 'was', 'being', 'who', 'very', 'well', 'your', 'beforehand', \"'s\", 'under', 'if', 'off', 'no', 'anyone', 'within', 'rather'}\n",
      "326\n"
     ]
    }
   ],
   "source": [
    "# Print spacy's stop word list\n",
    "print(nlp.Defaults.stop_words)\n",
    "print(len(nlp.Defaults.stop_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "608f9d2b",
   "metadata": {
    "papermill": {
     "duration": 0.00668,
     "end_time": "2023-02-20T20:24:03.804507",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.797827",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To remove stop words, we use the `.is_stop` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e8c7518f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:03.819951Z",
     "iopub.status.busy": "2023-02-20T20:24:03.819577Z",
     "iopub.status.idle": "2023-02-20T20:24:03.826090Z",
     "shell.execute_reply": "2023-02-20T20:24:03.824665Z"
    },
    "papermill": {
     "duration": 0.016892,
     "end_time": "2023-02-20T20:24:03.828342",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.811450",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['train', 'London', 'leaves', '10', 'Tuesday', '.']\n"
     ]
    }
   ],
   "source": [
    "# Stop word removal\n",
    "print([t.text for t in doc if not t.is_stop])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bc4924b",
   "metadata": {
    "papermill": {
     "duration": 0.006788,
     "end_time": "2023-02-20T20:24:03.843049",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.836261",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Depending on the application, you might want to **customize spacys** stop word list. This can be done as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "90cbfd3f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:03.858829Z",
     "iopub.status.busy": "2023-02-20T20:24:03.858356Z",
     "iopub.status.idle": "2023-02-20T20:24:03.864144Z",
     "shell.execute_reply": "2023-02-20T20:24:03.863010Z"
    },
    "papermill": {
     "duration": 0.016157,
     "end_time": "2023-02-20T20:24:03.866154",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.849997",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "nlp.Defaults.stop_words.add(\"ergo\")\n",
    "nlp.Defaults.stop_words.remove(\"whatever\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13af8d13",
   "metadata": {
    "papermill": {
     "duration": 0.006838,
     "end_time": "2023-02-20T20:24:03.880513",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.873675",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.3 Stemming</p>\n",
    "\n",
    "Stemming is the act of **reducing a word to its stem** by **removing suffixes** and sometimes prefixes depending on the language.\n",
    "\n",
    "For example, the words `\"developed\"` and `\"developing\"` both have the stem `\"develop\"`.\n",
    "\n",
    "While this technique also reduces the size of the vocabulary, it can result in **invalid words**, for example `\"studies\"` might be stemmed to `\"studi\"`. For this reason, stemming is rarely used these days. It turns out there is a **better altenative**, called **lemmatization**, which we'll look at next."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b1918b3",
   "metadata": {
    "papermill": {
     "duration": 0.007379,
     "end_time": "2023-02-20T20:24:03.895068",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.887689",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.4 Lemmatization</p>\n",
    "\n",
    "Lemmatization reduces a word down to its **lemma**, i.e. dictionary form. \n",
    "\n",
    "While this is similar to stemming, it also takes into account things like **tenses** and **synonyms**. For example, the words `\"did\"`, `\"done\"` and `\"doing\"` would be converted to the base form `\"do\"`.\n",
    "\n",
    "<center>\n",
    "<img src='../assets/2/lemmatization.jpg' width=600>\n",
    "</center>\n",
    "    \n",
    "It also takes into account whether a word is a **noun**, **verb** or **adjective** on deciding whether to lemmatize. For example, it might not modify some adjectives so not to change their meaning. (`\"energetic\"` is different to `\"energy\"`).\n",
    "\n",
    "Lemmatization is generally prefered to stemming because it is **more accurate and robust** while still offering the same benefit of vocabulary size reduction. It does however remove your ability to distinguish different **tenses**, which may be important for some applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7c1dbd84",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:03.910322Z",
     "iopub.status.busy": "2023-02-20T20:24:03.909975Z",
     "iopub.status.idle": "2023-02-20T20:24:03.919944Z",
     "shell.execute_reply": "2023-02-20T20:24:03.919082Z"
    },
    "papermill": {
     "duration": 0.020117,
     "end_time": "2023-02-20T20:24:03.922230",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.902113",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Tokenize\n",
    "s = \"She was the fastest swimmer.\"\n",
    "doc = nlp(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1770dfba",
   "metadata": {
    "papermill": {
     "duration": 0.006936,
     "end_time": "2023-02-20T20:24:03.936771",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.929835",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can view the lemmatization using the `.lemma_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "52f17617",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:03.952983Z",
     "iopub.status.busy": "2023-02-20T20:24:03.952581Z",
     "iopub.status.idle": "2023-02-20T20:24:03.958280Z",
     "shell.execute_reply": "2023-02-20T20:24:03.956955Z"
    },
    "papermill": {
     "duration": 0.016159,
     "end_time": "2023-02-20T20:24:03.960150",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.943991",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('She', 'she'), ('was', 'be'), ('the', 'the'), ('fastest', 'fast'), ('swimmer', 'swimmer'), ('.', '.')]\n"
     ]
    }
   ],
   "source": [
    "# Lemmatization\n",
    "print([(t.text,t.lemma_) for t in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cc45c1b",
   "metadata": {
    "papermill": {
     "duration": 0.006896,
     "end_time": "2023-02-20T20:24:03.974223",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.967327",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3. Advanced pre-processing\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.1 Part-of-speech tagging</p>\n",
    "\n",
    "Part-of-speech tagging is the method of **classifying how a word is used in a sentence**, for example, **noun, verb, adjective**.\n",
    "\n",
    "<center>\n",
    "<img src='../assets/2/pos-tagging.jpg' width=600>\n",
    "</center>\n",
    "\n",
    "This is very helpful because it can help us understand the **intent or action** of an ambiguous word. For example, when we say `\"Hand me a hammer.\"`, the word `\"hand\"` is a **verb** (doing word) as opposed to `\"The hammer is in my hand.\"` where it is a **noun** (thing) and has a different meaning. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4acc3d22",
   "metadata": {
    "papermill": {
     "duration": 0.006766,
     "end_time": "2023-02-20T20:24:03.989005",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.982239",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can access the part-of-speech tags using the `\".pos_\"` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "978b8302",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:04.005200Z",
     "iopub.status.busy": "2023-02-20T20:24:04.004838Z",
     "iopub.status.idle": "2023-02-20T20:24:04.009282Z",
     "shell.execute_reply": "2023-02-20T20:24:04.008636Z"
    },
    "papermill": {
     "duration": 0.015097,
     "end_time": "2023-02-20T20:24:04.010998",
     "exception": false,
     "start_time": "2023-02-20T20:24:03.995901",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('She', 'PRON'), ('was', 'AUX'), ('the', 'DET'), ('fastest', 'ADJ'), ('swimmer', 'NOUN'), ('.', 'PUNCT')]\n"
     ]
    }
   ],
   "source": [
    "# Part-of-speech\n",
    "print([(t.text,t.pos_) for t in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1f4c7e5",
   "metadata": {
    "papermill": {
     "duration": 0.007037,
     "end_time": "2023-02-20T20:24:04.025439",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.018402",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "A full description of the tags can be found using `\"spacy.explain\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "81f3c730",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:04.042655Z",
     "iopub.status.busy": "2023-02-20T20:24:04.042245Z",
     "iopub.status.idle": "2023-02-20T20:24:04.047678Z",
     "shell.execute_reply": "2023-02-20T20:24:04.046739Z"
    },
    "papermill": {
     "duration": 0.016146,
     "end_time": "2023-02-20T20:24:04.049557",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.033411",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('PRON', 'pronoun'), ('AUX', 'auxiliary'), ('DET', 'determiner'), ('ADJ', 'adjective'), ('NOUN', 'noun'), ('PUNCT', 'punctuation')]\n"
     ]
    }
   ],
   "source": [
    "print([(t.pos_,spacy.explain(t.pos_)) for t in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc39a22e",
   "metadata": {
    "papermill": {
     "duration": 0.006979,
     "end_time": "2023-02-20T20:24:04.063931",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.056952",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.2 Named Entity Recognition</p>\n",
    "\n",
    "Named Entity Recognition (NER) is the act of tagging **named entities** in text. \n",
    "\n",
    "A **named entity** is anything that can be referred to by a **proper name** and usually has the **proper noun** tag. Common examples include a person, cities, countries and companies. Note that it is common to extend entities to include money, time, dates, etc. \n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<center>\n",
    "<img src='../assets/2/ner.png' width=600>\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "\n",
    "NER can help **categorize and organize** a corpus. It is especially useful, for example, in helping **chatbots** raise accurate support tickets depending on the customer problem. \n",
    "\n",
    "Some of the **challenges** to building a state-of-the-art NER model include **type ambiguity**, where one word can have multiple meanings (e.g. Amazon - river or company?) and the fact that **entities can span multiple tokens** (e.g. John Smith). Luckily, spacy has very good NER model that we can utilize. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4a859756",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:04.079621Z",
     "iopub.status.busy": "2023-02-20T20:24:04.079275Z",
     "iopub.status.idle": "2023-02-20T20:24:04.090636Z",
     "shell.execute_reply": "2023-02-20T20:24:04.089692Z"
    },
    "papermill": {
     "duration": 0.021398,
     "end_time": "2023-02-20T20:24:04.092425",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.071027",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Tokenize\n",
    "s = \"Apple is looking at buying U.K. startup for $1 billion\"\n",
    "doc = nlp(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c187fc64",
   "metadata": {
    "papermill": {
     "duration": 0.006644,
     "end_time": "2023-02-20T20:24:04.106490",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.099846",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "There are two ways to do NER in spacy. The **first way** is via the `.ent_type_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c13bc910",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:04.123539Z",
     "iopub.status.busy": "2023-02-20T20:24:04.123177Z",
     "iopub.status.idle": "2023-02-20T20:24:04.127637Z",
     "shell.execute_reply": "2023-02-20T20:24:04.126772Z"
    },
    "papermill": {
     "duration": 0.015144,
     "end_time": "2023-02-20T20:24:04.129577",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.114433",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Apple', 'ORG'), ('is', ''), ('looking', ''), ('at', ''), ('buying', ''), ('U.K.', 'GPE'), ('startup', ''), ('for', ''), ('$', 'MONEY'), ('1', 'MONEY'), ('billion', 'MONEY')]\n"
     ]
    }
   ],
   "source": [
    "# Named Entity Recognition\n",
    "print([(t.text,t.ent_type_) for t in doc])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3b54a328",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:04.146171Z",
     "iopub.status.busy": "2023-02-20T20:24:04.145729Z",
     "iopub.status.idle": "2023-02-20T20:24:04.151350Z",
     "shell.execute_reply": "2023-02-20T20:24:04.150447Z"
    },
    "papermill": {
     "duration": 0.017617,
     "end_time": "2023-02-20T20:24:04.154558",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.136941",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Apple', 'ORG'), ('U.K.', 'GPE'), ('$', 'MONEY'), ('1', 'MONEY'), ('billion', 'MONEY')]\n"
     ]
    }
   ],
   "source": [
    "# Only print entities\n",
    "print([(t.text,t.ent_type_) for t in doc if t.ent_type != 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "933393d8",
   "metadata": {
    "papermill": {
     "duration": 0.006881,
     "end_time": "2023-02-20T20:24:04.169697",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.162816",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Like before, we can `spacy.explain` to understand each tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "75dc88cb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:04.185731Z",
     "iopub.status.busy": "2023-02-20T20:24:04.185319Z",
     "iopub.status.idle": "2023-02-20T20:24:04.192159Z",
     "shell.execute_reply": "2023-02-20T20:24:04.190181Z"
    },
    "papermill": {
     "duration": 0.017855,
     "end_time": "2023-02-20T20:24:04.194475",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.176620",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORG: Companies, agencies, institutions, etc.\n",
      "GPE: Countries, cities, states\n",
      "MONEY: Monetary values, including unit\n"
     ]
    }
   ],
   "source": [
    "# Entity explanation\n",
    "print('ORG:', spacy.explain('ORG'))\n",
    "print('GPE:', spacy.explain('GPE'))\n",
    "print('MONEY:', spacy.explain('MONEY'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1212d33",
   "metadata": {
    "papermill": {
     "duration": 0.008688,
     "end_time": "2023-02-20T20:24:04.212324",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.203636",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The **second way** to do NER in spacy is to use the `.ents` attribute. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "507af0b1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:04.230823Z",
     "iopub.status.busy": "2023-02-20T20:24:04.230211Z",
     "iopub.status.idle": "2023-02-20T20:24:04.237540Z",
     "shell.execute_reply": "2023-02-20T20:24:04.236541Z"
    },
    "papermill": {
     "duration": 0.019719,
     "end_time": "2023-02-20T20:24:04.239710",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.219991",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Apple', 'ORG'), ('U.K.', 'GPE'), ('$1 billion', 'MONEY')]\n"
     ]
    }
   ],
   "source": [
    "# Named Entity Recognition\n",
    "print([(ent.text, ent.label_) for ent in doc.ents])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28338539",
   "metadata": {
    "papermill": {
     "duration": 0.00763,
     "end_time": "2023-02-20T20:24:04.255886",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.248256",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Notice this time how `$1 billion` is grouped into **one entity**, whereas before each token was a separate entity."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3237eb81",
   "metadata": {
    "papermill": {
     "duration": 0.008002,
     "end_time": "2023-02-20T20:24:04.274476",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.266474",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Finally, we can **visualize** the entities using a spacy built-in function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2b055552",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:24:04.291305Z",
     "iopub.status.busy": "2023-02-20T20:24:04.290814Z",
     "iopub.status.idle": "2023-02-20T20:24:04.298793Z",
     "shell.execute_reply": "2023-02-20T20:24:04.298179Z"
    },
    "papermill": {
     "duration": 0.018516,
     "end_time": "2023-02-20T20:24:04.300438",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.281922",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Apple\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " is looking at buying \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    U.K.\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " startup for \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    $1 billion\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
       "</mark>\n",
       "</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import function\n",
    "from spacy import displacy\n",
    "\n",
    "# Visualize entities\n",
    "displacy.render(doc, style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c1de5ea",
   "metadata": {
    "papermill": {
     "duration": 0.007358,
     "end_time": "2023-02-20T20:24:04.315386",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.308028",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 4. Conclusion\n",
    "\n",
    "Whilst we have seen the most common pre-processing techniques in this notebook, there exist many more depending on the application. Some others ideas to keep in mind include converting **emoji's to text**, **language detection** in a mixed-language corpus, **spelling correction** and **parsing** intra-word relationships."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7fd1458",
   "metadata": {
    "papermill": {
     "duration": 0.007398,
     "end_time": "2023-02-20T20:24:04.330391",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.322993",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**References:**\n",
    "    \n",
    "* [NLP demystified](https://www.nlpdemystified.org/)\n",
    "\n",
    "### Coming UP\n",
    "#### [3. Bag Of Words and Similarity](./03_BOW_Similarity.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f86f89f",
   "metadata": {
    "papermill": {
     "duration": 0.007572,
     "end_time": "2023-02-20T20:24:04.346459",
     "exception": false,
     "start_time": "2023-02-20T20:24:04.338887",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Thanks for reading!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 23.529582,
   "end_time": "2023-02-20T20:24:07.185578",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-02-20T20:23:43.655996",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}