Notebooks/01_Tokenization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c97bc8cc",
   "metadata": {
    "papermill": {
     "duration": 0.007796,
     "end_time": "2023-02-20T20:20:32.774610",
     "exception": false,
     "start_time": "2023-02-20T20:20:32.766814",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<center>\n",
    "<img src=\"../assets/1/lang-pic.jpg\" width=600 align=\"center\">\n",
    "</center>\n",
    "    \n",
    "# 1. Introduction\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">1.1 NLP series</p>\n",
    "\n",
    "This is the **first in a series of notebooks** covering the **fundamentals of Natural Language Processing (NLP)**. I find that the best way to learn is by teaching others, hence why I am sharing my journey learning this field from scratch. I hope these notebooks can be helpful to you too. \n",
    "\n",
    "NLP series:\n",
    "\n",
    "1. **Tokenization** \n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/JUSTSUJAY/NLP_One_Shot/blob/main/Notebooks/01_Tokenization.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/JUSTSUJAY/NLP_One_Shot/blob/main/Notebooks/01_Tokenization.ipynb)\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">1.2 Outline</p>\n",
    "\n",
    "This notebook focuses on introducing and **motivating the field of NLP** from the very beginning. We will look at the **big picture** of what NLP is really about and also give an overview of common tasks. \n",
    "\n",
    "Then we will take the **first step** in any NLP problem, which is **tokenization**. This is the process of breaking down any large piece of unstructured text into **tokens**, usually words, that can then be further processed. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a93ebed",
   "metadata": {
    "papermill": {
     "duration": 0.006284,
     "end_time": "2023-02-20T20:20:32.787599",
     "exception": false,
     "start_time": "2023-02-20T20:20:32.781315",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2. Big picture\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.1 What is NLP?</p>\n",
    "\n",
    "<center>\n",
    "<img src='../assets/1/hello-banner.png' width=600>\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "**Natural Language Processing**, or NLP for short, is the study of **understanding and synthesizing** natural language using computers. It is a field that combines **mathematics**, **computer science** and **linguistics**, and has exploded in popularity in the last 10 years. \n",
    "\n",
    "Note: by **natural language**, we mean languages that have been **developed by humans** to communicate with each other, such as English, Spanish, Chinese, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e375d296",
   "metadata": {
    "papermill": {
     "duration": 0.006328,
     "end_time": "2023-02-20T20:20:32.800394",
     "exception": false,
     "start_time": "2023-02-20T20:20:32.794066",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.2 Common tasks</p>\n",
    "\n",
    "<img src='../assets/1/namedentity.png' width=250 style=\"float: right;\">\n",
    "\n",
    "Some of the **easier** NLP tasks include:\n",
    "* Spell checking\n",
    "* Keyword retrieval\n",
    "* Spam detection\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "<img src='../assets/1/wall-5.jpg' width=250 style=\"float: right;\">\n",
    "\n",
    "**Medium** difficulty NLP tasks include:\n",
    "* Sentiment analysis\n",
    "* Topic Modelling\n",
    "* Named entity recognition\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "<img src='../assets/1/chatbot2.png' width=250 style=\"float: right;\">\n",
    "\n",
    "**Hard** NLP tasks include:\n",
    "* Text summarisation\n",
    "* Machine translation\n",
    "* Conversational AI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdc3ad03",
   "metadata": {
    "papermill": {
     "duration": 0.006002,
     "end_time": "2023-02-20T20:20:32.812792",
     "exception": false,
     "start_time": "2023-02-20T20:20:32.806790",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.3 Why is NLP so hard?</p>\n",
    "\n",
    "Natural languages have been developed over **thousands of years** by the need for humans to **convey information** to each other **quickly**. But this information is often **complex** and **nuanced**; we speak differently depending on who we are talking to, information can be **assumed** based on present **context**/shared **memories** and **body language** helps present our **emotions** without words. \n",
    "\n",
    "There are so many **layers** to how humans interact (which we are still trying to fully understand ourselves) that there should be no surprise at how incredibly difficult it is to **teach computers** to do the same. \n",
    "\n",
    "<center>\n",
    "<img src='../assets/1/yaint-meme.png' width=300>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d809153",
   "metadata": {
    "papermill": {
     "duration": 0.006048,
     "end_time": "2023-02-20T20:20:32.825012",
     "exception": false,
     "start_time": "2023-02-20T20:20:32.818964",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Examples\n",
    "\n",
    "<br>\n",
    "\n",
    "* > \"I went to the bank.\"\n",
    "\n",
    "Is this a financial bank or the river bank? This is an example of **lexical ambiguity**, i.e. when there are multiple meanings within a single word. \n",
    "\n",
    "<br>\n",
    "\n",
    "* > \"Call me a taxi.\"\n",
    "\n",
    "Should our response be \"Hi taxi\" or \"Sure, I'll call the taxi company\". This is obvious to a human but not so much to a computer and is called **syntactic ambiguity**, i.e. when there are multiple meanings within a sentence. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6244b58",
   "metadata": {
    "papermill": {
     "duration": 0.007543,
     "end_time": "2023-02-20T20:20:32.838759",
     "exception": false,
     "start_time": "2023-02-20T20:20:32.831216",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3. Tokenization\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.1 What is tokenization?</p>\n",
    "\n",
    "We usually start an NLP project with a large body of text, called a **corpus**. This could be a collection of tweets, website reviews or transcriptions of films, for example. We need to **pre-process** our corpus to give it enough **structure** to be used in a machine learning model and tokenization is the most common first step.\n",
    "\n",
    "**Tokenization** is the process of breaking down a corpus into **tokens**. The procedure might look like **segmenting** a piece of text into sentences and then further segmenting these sentences into individual **words, numbers and punctuation**, which would be tokens. \n",
    "\n",
    "<center>\n",
    "<img src='../assets/1/tokenization2.jpg' width=600>\n",
    "</center>\n",
    "\n",
    "Each token should be chosen to be as **small as possible** while still carrying carrying **meaning on its own**. For example, `\"£10\"` can be split into the two tokens `\"£\"` and `\"10\"` as each one possess its own meaning. \n",
    "\n",
    "Note that researches are still trying to find out the best way to tokenize text. There exist effective models that break down words into smaller parts like splitting `\"running\"` into `[\"run\",\"-ing\"]` (called morphemes), or even into individual letters `[\"r\", \"u\", \"n\", \"n\", \"i\", \"n\", \"g\"]` (called graphemes). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aec042c",
   "metadata": {
    "papermill": {
     "duration": 0.006048,
     "end_time": "2023-02-20T20:20:32.851085",
     "exception": false,
     "start_time": "2023-02-20T20:20:32.845037",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.2 Tokenization using spaCy</p>\n",
    "\n",
    "<center>\n",
    "<img src='../assets/1/spacy.png' width=250>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "Lucky for us, there exist **robust NLP libraries** which can perform tokenization for us. Let's see how to do this with one of the most popular ones called **spaCy**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e83705d6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:32.865980Z",
     "iopub.status.busy": "2023-02-20T20:20:32.865206Z",
     "iopub.status.idle": "2023-02-20T20:20:43.728077Z",
     "shell.execute_reply": "2023-02-20T20:20:43.726965Z"
    },
    "papermill": {
     "duration": 10.873867,
     "end_time": "2023-02-20T20:20:43.731187",
     "exception": false,
     "start_time": "2023-02-20T20:20:32.857320",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spacy 3.3.1\n"
     ]
    }
   ],
   "source": [
    "# Import spacy library\n",
    "import spacy\n",
    "print(spacy.__name__, spacy.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a36cbeb8",
   "metadata": {
    "papermill": {
     "duration": 0.006698,
     "end_time": "2023-02-20T20:20:43.744895",
     "exception": false,
     "start_time": "2023-02-20T20:20:43.738197",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Next we load a **statistical model** of the English language **trained on web articles**. Its capabilities include tokenization, among other things. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2ee0f3cc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:43.761732Z",
     "iopub.status.busy": "2023-02-20T20:20:43.761057Z",
     "iopub.status.idle": "2023-02-20T20:20:44.627501Z",
     "shell.execute_reply": "2023-02-20T20:20:44.626625Z"
    },
    "papermill": {
     "duration": 0.877281,
     "end_time": "2023-02-20T20:20:44.629934",
     "exception": false,
     "start_time": "2023-02-20T20:20:43.752653",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load the spacy model\n",
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0185ab95",
   "metadata": {
    "papermill": {
     "duration": 0.006503,
     "end_time": "2023-02-20T20:20:44.644172",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.637669",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To tokenize a string, we simply pass it in as an **argument** to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d570e15b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:44.659524Z",
     "iopub.status.busy": "2023-02-20T20:20:44.658837Z",
     "iopub.status.idle": "2023-02-20T20:20:44.684921Z",
     "shell.execute_reply": "2023-02-20T20:20:44.683694Z"
    },
    "papermill": {
     "duration": 0.036828,
     "end_time": "2023-02-20T20:20:44.687688",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.650860",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Tokenize string\n",
    "s = \"Noah doesn't like to run when it rains.\"\n",
    "doc = nlp(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45a1d7a5",
   "metadata": {
    "papermill": {
     "duration": 0.006566,
     "end_time": "2023-02-20T20:20:44.701114",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.694548",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "And we can **view the tokens** using one line of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "015d90b7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:44.716104Z",
     "iopub.status.busy": "2023-02-20T20:20:44.715700Z",
     "iopub.status.idle": "2023-02-20T20:20:44.720771Z",
     "shell.execute_reply": "2023-02-20T20:20:44.719709Z"
    },
    "papermill": {
     "duration": 0.015242,
     "end_time": "2023-02-20T20:20:44.723024",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.707782",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Noah', 'does', \"n't\", 'like', 'to', 'run', 'when', 'it', 'rains', '.']\n"
     ]
    }
   ],
   "source": [
    "# Print tokens\n",
    "print([t.text for t in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e819481",
   "metadata": {
    "papermill": {
     "duration": 0.006776,
     "end_time": "2023-02-20T20:20:44.736882",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.730106",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "*Notes:*\n",
    "* `\"doesn't\"` gets split into two tokens: `\"does\"` and `\"n't\"`.\n",
    "* the full stop `\".\"` gets its own token."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f981bf7",
   "metadata": {
    "papermill": {
     "duration": 0.00631,
     "end_time": "2023-02-20T20:20:44.749934",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.743624",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.3 Types and attributes</p>\n",
    "\n",
    "The `doc` object is a **container**, which can be indexed and sliced like a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ab5ba46e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:44.764952Z",
     "iopub.status.busy": "2023-02-20T20:20:44.764541Z",
     "iopub.status.idle": "2023-02-20T20:20:44.770424Z",
     "shell.execute_reply": "2023-02-20T20:20:44.769307Z"
    },
    "papermill": {
     "duration": 0.017073,
     "end_time": "2023-02-20T20:20:44.773552",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.756479",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Noah\n",
      "Noah doesn't\n"
     ]
    }
   ],
   "source": [
    "# Index and slice example\n",
    "print(doc[0])\n",
    "print(doc[0:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "768d0e47",
   "metadata": {
    "papermill": {
     "duration": 0.006587,
     "end_time": "2023-02-20T20:20:44.788034",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.781447",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Each entry in the doc object is a **token object**. And if you slice a doc object you get a **span object**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4d03fd31",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:44.803611Z",
     "iopub.status.busy": "2023-02-20T20:20:44.802900Z",
     "iopub.status.idle": "2023-02-20T20:20:44.809036Z",
     "shell.execute_reply": "2023-02-20T20:20:44.807658Z"
    },
    "papermill": {
     "duration": 0.016338,
     "end_time": "2023-02-20T20:20:44.811050",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.794712",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'spacy.tokens.doc.Doc'>\n",
      "<class 'spacy.tokens.token.Token'>\n",
      "<class 'spacy.tokens.span.Span'>\n"
     ]
    }
   ],
   "source": [
    "# Object types\n",
    "print(type(doc))\n",
    "print(type(doc[0]))\n",
    "print(type(doc[0:3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce2daaae",
   "metadata": {
    "papermill": {
     "duration": 0.006595,
     "end_time": "2023-02-20T20:20:44.824432",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.817837",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Each token has several **attributes** such as language, length, index, etc. We will explore these in more detail in later notebooks but here are some examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "191910ae",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:44.839834Z",
     "iopub.status.busy": "2023-02-20T20:20:44.839426Z",
     "iopub.status.idle": "2023-02-20T20:20:44.845510Z",
     "shell.execute_reply": "2023-02-20T20:20:44.844293Z"
    },
    "papermill": {
     "duration": 0.017044,
     "end_time": "2023-02-20T20:20:44.848350",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.831306",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "like\n",
      "en\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "# Token attribute examples\n",
    "print(doc[3].text)\n",
    "print(doc[3].lang_)\n",
    "print(doc[3].__len__())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f417b85d",
   "metadata": {
    "papermill": {
     "duration": 0.006658,
     "end_time": "2023-02-20T20:20:44.862844",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.856186",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can locate the index of each token using the `.i` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3f06524a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:44.878994Z",
     "iopub.status.busy": "2023-02-20T20:20:44.877931Z",
     "iopub.status.idle": "2023-02-20T20:20:44.884044Z",
     "shell.execute_reply": "2023-02-20T20:20:44.882784Z"
    },
    "papermill": {
     "duration": 0.016713,
     "end_time": "2023-02-20T20:20:44.886440",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.869727",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Noah', 0), ('does', 1), (\"n't\", 2), ('like', 3), ('to', 4), ('run', 5)]\n"
     ]
    }
   ],
   "source": [
    "# Locate index of tokens\n",
    "print([(t.text, t.i) for t in doc[:6]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e353c23d",
   "metadata": {
    "papermill": {
     "duration": 0.006798,
     "end_time": "2023-02-20T20:20:44.900350",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.893552",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.4 Tokenizing paragraphs</p>\n",
    "\n",
    "We can tokenize paragraphs in a very similar way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "750c9295",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:44.916473Z",
     "iopub.status.busy": "2023-02-20T20:20:44.915703Z",
     "iopub.status.idle": "2023-02-20T20:20:44.930218Z",
     "shell.execute_reply": "2023-02-20T20:20:44.929104Z"
    },
    "papermill": {
     "duration": 0.025624,
     "end_time": "2023-02-20T20:20:44.933028",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.907404",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Tokenize multiple sentences\n",
    "s = \"Hello there! General Kenobi. You are a bold one.\"\n",
    "doc = nlp(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b32954e",
   "metadata": {
    "papermill": {
     "duration": 0.006835,
     "end_time": "2023-02-20T20:20:44.947023",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.940188",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can iterate through the sentences using the `.sents` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1a5a1bcf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:44.963480Z",
     "iopub.status.busy": "2023-02-20T20:20:44.963059Z",
     "iopub.status.idle": "2023-02-20T20:20:44.971697Z",
     "shell.execute_reply": "2023-02-20T20:20:44.970491Z"
    },
    "papermill": {
     "duration": 0.019677,
     "end_time": "2023-02-20T20:20:44.974143",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.954466",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Hello there!, General Kenobi., You are a bold one.]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print sentences\n",
    "list(doc.sents)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2a0e63c",
   "metadata": {
    "papermill": {
     "duration": 0.006872,
     "end_time": "2023-02-20T20:20:44.988203",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.981331",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Note that each sentence is a **span object** of the original document. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "00d6924e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:45.004308Z",
     "iopub.status.busy": "2023-02-20T20:20:45.003886Z",
     "iopub.status.idle": "2023-02-20T20:20:45.010466Z",
     "shell.execute_reply": "2023-02-20T20:20:45.009700Z"
    },
    "papermill": {
     "duration": 0.017075,
     "end_time": "2023-02-20T20:20:45.012495",
     "exception": false,
     "start_time": "2023-02-20T20:20:44.995420",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "spacy.tokens.span.Span"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Object type\n",
    "type(list(doc.sents)[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81fcfac7",
   "metadata": {
    "papermill": {
     "duration": 0.007047,
     "end_time": "2023-02-20T20:20:45.026860",
     "exception": false,
     "start_time": "2023-02-20T20:20:45.019813",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can print each token exactly the **same as before**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "399ba14c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:20:45.043221Z",
     "iopub.status.busy": "2023-02-20T20:20:45.042796Z",
     "iopub.status.idle": "2023-02-20T20:20:45.048520Z",
     "shell.execute_reply": "2023-02-20T20:20:45.047344Z"
    },
    "papermill": {
     "duration": 0.016879,
     "end_time": "2023-02-20T20:20:45.050877",
     "exception": false,
     "start_time": "2023-02-20T20:20:45.033998",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hello', 'there', '!', 'General', 'Kenobi', '.', 'You', 'are', 'a', 'bold', 'one', '.']\n"
     ]
    }
   ],
   "source": [
    "# Print tokens\n",
    "print([t.text for t in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79b679b6",
   "metadata": {
    "papermill": {
     "duration": 0.006989,
     "end_time": "2023-02-20T20:20:45.065230",
     "exception": false,
     "start_time": "2023-02-20T20:20:45.058241",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**References:**\n",
    "* [NLP demystified](https://www.nlpdemystified.org/)\n",
    "\n",
    "### Coming UP\n",
    "#### [2. Text Preprocessing](./02_Pre_Processing.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc9e1fb5",
   "metadata": {
    "papermill": {
     "duration": 0.007714,
     "end_time": "2023-02-20T20:20:45.080509",
     "exception": false,
     "start_time": "2023-02-20T20:20:45.072795",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Thanks for reading!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 24.594495,
   "end_time": "2023-02-20T20:20:48.079972",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-02-20T20:20:23.485477",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}