Named_Entity_Recognition_with_NLTK_and_SpaCy.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Named Entity Recognition with NLTK and SpaCy",
      "provenance": [],
      "authorship_tag": "ABX9TyNCJ042vVJp/5px1tP+mDFc",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hassaanhameed786/NLP/blob/main/Named_Entity_Recognition_with_NLTK_and_SpaCy.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Named entity recognition (NER)is probably the first step towards information extraction that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. NER is used in many fields in Natural Language Processing (NLP), and it can help answering many real-world questions, such as:\n",
        "Which companies were mentioned in the news article?\n",
        "Were specified products mentioned in complaints or reviews?\n",
        "Does the tweet contain the name of a person? Does the tweet contain this person’s location?"
      ],
      "metadata": {
        "id": "lKQ2fPmWmaiF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "Wr_ryciQmXv2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('punkt')\n",
        "from nltk.tokenize import word_tokenize\n",
        "from nltk.tag import pos_tag"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QB_he01ImRCs",
        "outputId": "94c56b5b-e882-4e50-8bec-1e519b86d90a"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "rO0o8R9RmPfu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# random sentence from the goolgle news \n",
        "ex = 'An 11-year-old who survived the Texas school shooting is scared the gunman is still is out to get her, her parents say'"
      ],
      "metadata": {
        "id": "LpgcTv1vmfHQ"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# # Data Preprocessing"
      ],
      "metadata": {
        "id": "E1pA8tZzmkRI"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "ZDcOqbVHlugY"
      },
      "outputs": [],
      "source": [
        "# apply word tokenization and part-of-speech tagging to the sentence\n",
        "def preprocess(sent):\n",
        "    sent = nltk.word_tokenize(sent)\n",
        "    sent = nltk.pos_tag(sent)\n",
        "    return sent"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "nltk.download('averaged_perceptron_tagger')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aMkEMfESmptd",
        "outputId": "abe8029f-ca97-49ff-a3f2-953d74be1a2c"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sent = preprocess(ex)\n",
        "sent"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "347puIW6msvF",
        "outputId": "20dee803-d53e-4e73-b68a-8d666d9595c2"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('An', 'DT'),\n",
              " ('11-year-old', 'JJ'),\n",
              " ('who', 'WP'),\n",
              " ('survived', 'VBD'),\n",
              " ('the', 'DT'),\n",
              " ('Texas', 'NNP'),\n",
              " ('school', 'NN'),\n",
              " ('shooting', 'NN'),\n",
              " ('is', 'VBZ'),\n",
              " ('scared', 'VBN'),\n",
              " ('the', 'DT'),\n",
              " ('gunman', 'NN'),\n",
              " ('is', 'VBZ'),\n",
              " ('still', 'RB'),\n",
              " ('is', 'VBZ'),\n",
              " ('out', 'RP'),\n",
              " ('to', 'TO'),\n",
              " ('get', 'VB'),\n",
              " ('her', 'PRP'),\n",
              " (',', ','),\n",
              " ('her', 'PRP$'),\n",
              " ('parents', 'NNS'),\n",
              " ('say', 'VBP')]"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## implement noun phrase chunking"
      ],
      "metadata": {
        "id": "qYWua8ENmxHa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Our chunk pattern consists of one rule, that a noun phrase, NP\n",
        "# it should be formed whenever the chunker finds an optional determiner, DT\n",
        "#  followed by any number of adjectives, JJ, and then a noun, NN.\n",
        "\n",
        "pattern = 'NP: {<DT>?<JJ>*<NN>}'"
      ],
      "metadata": {
        "id": "b9N9y0sNmtny"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Chunking"
      ],
      "metadata": {
        "id": "4uchAOlem3br"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cp = nltk.RegexpParser(pattern)\n",
        "cs = cp.parse(sent)\n",
        "print(cs) ## output can be read as a tree "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8jVn5fLqmtlG",
        "outputId": "046cdf21-4aa7-49b9-f145-98b1937680e4"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(S\n",
            "  An/DT\n",
            "  11-year-old/JJ\n",
            "  who/WP\n",
            "  survived/VBD\n",
            "  the/DT\n",
            "  Texas/NNP\n",
            "  (NP school/NN)\n",
            "  (NP shooting/NN)\n",
            "  is/VBZ\n",
            "  scared/VBN\n",
            "  (NP the/DT gunman/NN)\n",
            "  is/VBZ\n",
            "  still/RB\n",
            "  is/VBZ\n",
            "  out/RP\n",
            "  to/TO\n",
            "  get/VB\n",
            "  her/PRP\n",
            "  ,/,\n",
            "  her/PRP$\n",
            "  parents/NNS\n",
            "  say/VBP)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "IOB tags have become the standard way to represent chunk structures in files, and we will also be using this format."
      ],
      "metadata": {
        "id": "-mN5gXkEm9SD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "Rl42U7ijnDY6"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk.chunk import conlltags2tree, tree2conlltags\n",
        "from pprint import pprint\n",
        "iob_tagged = tree2conlltags(cs)\n",
        "pprint(iob_tagged)\n",
        "ne_tree"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 599
        },
        "id": "ceVc48hTmtiQ",
        "outputId": "c15e41d3-3b72-496e-8d0e-3171cedc0bb0"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('An', 'DT', 'O'),\n",
            " ('11-year-old', 'JJ', 'O'),\n",
            " ('who', 'WP', 'O'),\n",
            " ('survived', 'VBD', 'O'),\n",
            " ('the', 'DT', 'O'),\n",
            " ('Texas', 'NNP', 'O'),\n",
            " ('school', 'NN', 'B-NP'),\n",
            " ('shooting', 'NN', 'B-NP'),\n",
            " ('is', 'VBZ', 'O'),\n",
            " ('scared', 'VBN', 'O'),\n",
            " ('the', 'DT', 'B-NP'),\n",
            " ('gunman', 'NN', 'I-NP'),\n",
            " ('is', 'VBZ', 'O'),\n",
            " ('still', 'RB', 'O'),\n",
            " ('is', 'VBZ', 'O'),\n",
            " ('out', 'RP', 'O'),\n",
            " ('to', 'TO', 'O'),\n",
            " ('get', 'VB', 'O'),\n",
            " ('her', 'PRP', 'O'),\n",
            " (',', ',', 'O'),\n",
            " ('her', 'PRP$', 'O'),\n",
            " ('parents', 'NNS', 'O'),\n",
            " ('say', 'VBP', 'O')]\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-13-3d0eec511616>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0miob_tagged\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtree2conlltags\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mpprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miob_tagged\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mne_tree\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'ne_tree' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        " nltk.download('maxent_ne_chunker')\n",
        " nltk.download('words')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lL7J0TiLnQpk",
        "outputId": "58c8e5af-1ec3-41d7-e046-7344ec89eabc"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package maxent_ne_chunker to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Package maxent_ne_chunker is already up-to-date!\n",
            "[nltk_data] Downloading package words to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/words.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk import ne_chunk\n",
        "ne_tree = ne_chunk(pos_tag(word_tokenize(ex)))\n",
        "print(ne_tree)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pvxchvmCmss5",
        "outputId": "7d52afa9-0e29-48e3-c1a7-a827fc7240c5"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(S\n",
            "  An/DT\n",
            "  11-year-old/JJ\n",
            "  who/WP\n",
            "  survived/VBD\n",
            "  the/DT\n",
            "  (GPE Texas/NNP)\n",
            "  school/NN\n",
            "  shooting/NN\n",
            "  is/VBZ\n",
            "  scared/VBN\n",
            "  the/DT\n",
            "  gunman/NN\n",
            "  is/VBZ\n",
            "  still/RB\n",
            "  is/VBZ\n",
            "  out/RP\n",
            "  to/TO\n",
            "  get/VB\n",
            "  her/PRP\n",
            "  ,/,\n",
            "  her/PRP$\n",
            "  parents/NNS\n",
            "  say/VBP)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# using spacy help this\n",
        "import spacy\n",
        "from spacy import displacy\n",
        "from collections import Counter\n",
        "import en_core_web_sm\n",
        "nlp = en_core_web_sm.load()"
      ],
      "metadata": {
        "id": "e0gT-Y45msqj"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**one of the nice things about Spacy is that we only need to apply nlp once, the entire background pipeline will return the objects.**"
      ],
      "metadata": {
        "id": "TLUWIZ_pn-31"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "doc = nlp('An 11-year-old who survived the Texas school shooting is scared the gunman is still is out to get her, her parents say')"
      ],
      "metadata": {
        "id": "SxQ4-YkDmsh9"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pprint([(X.text, X.label_) for X in doc.ents])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OHaIACuymsWl",
        "outputId": "3d9fd821-83d9-4b6f-fc76-72663d626a6f"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('Texas', 'GPE')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Token\n",
        "During the above example, we were working on entity level, in the following example, we are demonstrating token-level entity annotation using the BILUO tagging scheme to describe the entity boundaries."
      ],
      "metadata": {
        "id": "H-K4f59PoO8I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pprint([(X, X.ent_iob_, X.ent_type_) for X in doc])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iY72CykOoQIr",
        "outputId": "3366917c-ddcd-489a-ddde-8480f9e926ba"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[(An, 'O', ''),\n",
            " (11-year, 'O', ''),\n",
            " (-, 'O', ''),\n",
            " (old, 'O', ''),\n",
            " (who, 'O', ''),\n",
            " (survived, 'O', ''),\n",
            " (the, 'O', ''),\n",
            " (Texas, 'B', 'GPE'),\n",
            " (school, 'O', ''),\n",
            " (shooting, 'O', ''),\n",
            " (is, 'O', ''),\n",
            " (scared, 'O', ''),\n",
            " (the, 'O', ''),\n",
            " (gunman, 'O', ''),\n",
            " (is, 'O', ''),\n",
            " (still, 'O', ''),\n",
            " (is, 'O', ''),\n",
            " (out, 'O', ''),\n",
            " (to, 'O', ''),\n",
            " (get, 'O', ''),\n",
            " (her, 'O', ''),\n",
            " (,, 'O', ''),\n",
            " (her, 'O', ''),\n",
            " (parents, 'O', ''),\n",
            " (say, 'O', '')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\"B\" means the token begins an entity, \"I\" means it is inside an entity, \"O\" means it is outside an entity, and \"\" means no entity tag is set."
      ],
      "metadata": {
        "id": "v9TVGx4goah2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "--------------------------------------------------------------------------------\n",
        "\n"
      ],
      "metadata": {
        "id": "C2v1hPPjpMsN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "6Mbx4Y2spXXu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Extracting named entity from an article \n",
        "\n",
        "Now let’s get serious with SpaCy and extracting named entities from article https://www.msn.com/en-us/news/world/russian-advances-in-ukraine-s-east-mark-a-tipping-point/ar-AAXUdDc?ocid=winp1taskbar&cvid=0a0edfc299cc4fa1995385eee0310321"
      ],
      "metadata": {
        "id": "uadF4x4iofcO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "dY5KEn1KpZGi"
      },
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "_hnzfi5QpZAa"
      },
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "Dvtk1h-7pY9u"
      },
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from bs4 import BeautifulSoup\n",
        "import requests\n",
        "import re\n",
        "\n",
        "def url_to_string(url):\n",
        "    res = requests.get(url)\n",
        "    html = res.text\n",
        "    soup = BeautifulSoup(html, 'html5lib')\n",
        "    for script in soup([\"script\", \"style\", 'aside']):\n",
        "        script.extract()\n",
        "        return \" \".join(re.split(r'[\\n\\t]+', soup.get_text()))\n",
        "\n",
        "\n",
        "\n",
        "ny_bb = url_to_string('https://propakistani.pk/2022/05/31/fbr-posts-officers-at-directorate-general-of-dnfbps-under-fatf-requirement/')\n",
        "article = nlp(ny_bb)\n",
        "len(article.ents)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SwRaMLfvobFx",
        "outputId": "60a2d0f7-a01c-401a-93e5-29170eb8f1f5"
      },
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "41"
            ]
          },
          "metadata": {},
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "labels = [x.label_ for x in article.ents]\n",
        "Counter(labels)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KaFTDPj6p4rI",
        "outputId": "8739d776-96eb-4bfc-c187-a6c0f99b4079"
      },
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Counter({'CARDINAL': 11,\n",
              "         'DATE': 1,\n",
              "         'MONEY': 8,\n",
              "         'ORDINAL': 1,\n",
              "         'ORG': 3,\n",
              "         'PERCENT': 6,\n",
              "         'PERSON': 5,\n",
              "         'PRODUCT': 1,\n",
              "         'WORK_OF_ART': 5})"
            ]
          },
          "metadata": {},
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "items = [x.text for x in article.ents]\n",
        "Counter(items).most_common(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LC29gpsRp-D_",
        "outputId": "fd9b7616-431f-403c-b36d-02555c477106"
      },
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('0.2', 4), ('#cf-bubbles', 4), ('#cf', 2)]"
            ]
          },
          "metadata": {},
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = [x for x in article.sents]\n",
        "print(sentences[20])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pAQ32K7BqBMq",
        "outputId": "e3642988-7de8-45e6-9dc6-beab9bff3e0f"
      },
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            ":nth-child(2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "displacy.render(nlp(str(sentences[20])), jupyter=True, style='ent')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "id": "rbxbDgIpqG8S",
        "outputId": "c4f3aa86-48c1-498f-8f87-d66c172e7697"
      },
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">:\n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    nth-child(2\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              ")</div></span>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "displacy.render(nlp(str(sentences[20])), style='dep', jupyter = True, options = {'distance': 120})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 278
        },
        "id": "Ak62hVMnqG5c",
        "outputId": "51137d7a-c572-4deb-b72e-efbb845e9f79"
      },
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"2612da5e5c4e46ee9d9cd821aaddcb5c-0\" class=\"displacy\" width=\"410\" height=\"257.0\" direction=\"ltr\" style=\"max-width: none; height: 257.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"167.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">:</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PUNCT</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"167.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"170\">nth-</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"170\">PROPN</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"167.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"290\">child(2)</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"290\">NOUN</tspan>\n",
              "</text>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-2612da5e5c4e46ee9d9cd821aaddcb5c-0-0\" stroke-width=\"2px\" d=\"M70,122.0 C70,2.0 290.0,2.0 290.0,122.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-2612da5e5c4e46ee9d9cd821aaddcb5c-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">punct</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M70,124.0 L62,112.0 78,112.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-2612da5e5c4e46ee9d9cd821aaddcb5c-0-1\" stroke-width=\"2px\" d=\"M190,122.0 C190,62.0 285.0,62.0 285.0,122.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-2612da5e5c4e46ee9d9cd821aaddcb5c-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M190,124.0 L182,112.0 198,112.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "</svg></span>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "[(x.orth_,x.pos_, x.lemma_) for x in [y \n",
        "                                      for y\n",
        "                                      in nlp(str(sentences[20])) \n",
        "                                      if not y.is_stop and y.pos_ != 'PUNCT']]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1LwBMHFlqGt-",
        "outputId": "4274987b-bba2-4902-c985-b8bf2a3f6006"
      },
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('nth', 'PROPN', 'nth'), ('child(2', 'NOUN', 'child(2')]"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dict([(str(x), x.label_) for x in nlp(str(sentences[20])).ents])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-s4f_v4zqnxk",
        "outputId": "807759c7-c033-4184-b5e8-19f39a07281c"
      },
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'nth-child(2': 'ORG'}"
            ]
          },
          "metadata": {},
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print([(x, x.ent_iob_, x.ent_type_) for x in sentences[20]])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U1Ljz3Taqnul",
        "outputId": "1b25fb8b-beeb-434a-da88-7bb45ad3a95a"
      },
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[(:, 'O', ''), (nth, 'O', ''), (-, 'O', ''), (child(2, 'O', ''), (), 'O', '')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "ghWqjTz9rEzh"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}