Practical_5_POS_Tagging.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00000-fd0e1d9d-3f5a-4db2-b7dc-e4becf5b3724",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "# Part-of-Speech Tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00001-5a9607e8-97c0-4f78-acb7-070cb5950807",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "**Part of speech** (POS, syntactic category, word class): a category of words that have similar grammatical properties and display similar syntactic behavior (they play similar roles within the grammatical structure of sentences) and sometimes similar morphology."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00002-1c0307bf-5af4-4454-ae66-23f4a2e34feb",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "**Less official desciption**\n",
    "\n",
    " Words that somehow ‘behave’ alike:\n",
    " \n",
    "- Appear in similar contexts \n",
    "- Perform similar functions in sentences \n",
    "- Undergo similar transformations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00002-ae94d152-40ee-4f1b-8ea7-574a492a4b72",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Wikipedia:\n",
    "- **Noun** (names): a word or lexical item denoting any abstract (abstract noun: e.g. home) or concrete entity (concrete noun: e.g. house); a person (police officer, Michael), place (coastline, London), thing (necktie, television), idea (happiness), or quality (bravery). Nouns can also be classified as count nouns or non-count nouns; some can belong to either category. The most common part of speech; they are called naming words.\n",
    "\n",
    "Important: **proper nouns** (names of specific perons or entities. in English usually not preceded by articles - e.g. Anne, IBM, London) VS common nouns \n",
    "Pronoun (replaces or places again): a substitute for a noun or noun phrase (them, he). Pronouns make sentences shorter and clearer since they replace nouns.\n",
    "- **Adjective** (describes, limits): a modifier of a noun or pronoun (big, brave). Adjectives make the meaning of another word (noun) more precise.\n",
    "- **Verb** (states action or being): a word denoting an action (walk), occurrence (happen), or state of being (be). Without a verb a group of words cannot be a clause or sentence.\n",
    "- **Adverb** (describes, limits): a modifier of an adjective, verb, or another adverb (very, quite). Adverbs make language more precise.\n",
    "- **Preposition** (relates): a word that relates words to each other in a phrase or sentence and aids in syntactic context (in, of). Prepositions show the relationship between a noun or a pronoun with another word in the sentence.\n",
    "- **Conjunction** (connects): a syntactic connector; links words, phrases, or clauses (and, but). Conjunctions connect words or group of words\n",
    "- **Interjection** (expresses feelings and emotions): an emotional greeting or exclamation (Huzzah, Alas). Interjections express strong feelings and emotions.\n",
    "- **Article** (describes, limits): a grammatical marker of definiteness (the) or indefiniteness (a, an). The article is not always listed among the parts of speech. It is considered by some grammarians to be a type of adjective or sometimes the term 'determiner' (a broader class) is used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00004-dcd2bd0b-5430-42eb-bf76-0b9599234f27",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00004-3b91aade-03ca-4f4e-9a88-153b079f4264",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Why is it **useful (in NLP)** to know what POS a word belongs to? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00004-6b3606d8-d1bb-4d43-bbe3-3991db4ed427",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "- **parsing** (POS defines the neighbouring words of the keyword: e.g. in English nouns are preceded by determiners and articles)\n",
    "\n",
    "- **name entity recognition** in information extraction (e.g. find people or organisation names)\n",
    "- **coreference resolution**\n",
    "- **speech recognition/synthesis** (e.g. CONtent or conTENT - pronounciation depend on the part of speech!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00006-069f02e2-7a8f-44a2-98bb-12ff50112433",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "### Two main classes [3]\n",
    "\n",
    "- **Closed class**\n",
    "\n",
    "    - Prepositions: of, in, by, ...\n",
    "    - Auxiliaries: may, can, will had, been, ... \n",
    "    - Pronouns: I, you, she, mine, his, them, ... \n",
    "    - Usually function words (short common words which play a role in grammar)\n",
    "\n",
    "- **Open class (English has 4 of them)**\n",
    "    - Nouns, \n",
    "    - Verbs\n",
    "    - Adjectives\n",
    "    - Adverbs\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00005-2ebad921-eb2a-4a43-ac70-2264f842df04",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "In NLP, parts of speech are usually defined not by the semantics of the class  (e.g. nouns denote people and things and verbs denote actions - normally!) - but raher by their morphological and syntactic properties. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00006-9f8a3b8a-adb1-4226-9d4f-afcca0385061",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "- **Morphology**: the study of words, how they are formed, and their relationship to other words in the same language. It analyzes the structure of words and parts of words, such as stems, root words, prefixes, and suffixes.\n",
    "- **Syntax**: the set of rules, principles, and processes that govern the structure of sentences (sentence structure) in a given language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00008-bdc0cabc-652f-40d4-8973-6ff421327154",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "# POS tagger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00009-25e5d13d-bfe2-432b-bbee-59977dab8862",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "A part-of-speech tagger, or POS-tagger, processes a sequence of words and attaches a part of speech tag to each word.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00010-c2a45816-21ea-4a69-9786-8298a93861d7",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "## NLTK POS tagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00012-e31d982f-6620-4086-ae17-ef0b0eb47ff8",
    "deepnote_cell_type": "code",
    "execution_millis": 3769,
    "execution_start": 1603806231654,
    "output_cleared": false,
    "source_hash": "91d56118",
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install nltk\n",
    "import nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "nltk.download('punkt')\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "nltk.download('tagsets')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00010-4aaf6096-dcf4-4926-8e28-74e795a7102d",
    "deepnote_cell_type": "code",
    "execution_millis": 7,
    "execution_start": 1603810153729,
    "output_cleared": false,
    "source_hash": "61a6ad8f",
    "tags": []
   },
   "outputs": [],
   "source": [
    "text = word_tokenize(\"Göran Hansson, secretary general of the Royal Swedish Academy of Sciences, tells Science that some of his colleagues have been hounded for expressing doubt about the country’s lax pandemic policies\")\n",
    "nltk.pos_tag(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00013-fbf8adf8-27e5-4183-a5dc-5f3e9f4e4a98",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "How to find out what the tags mean? \n",
    "There is Penn Treebank Tagset https://www.eecis.udel.edu/~vijay/cis889/ie/pos-set.pdf \n",
    "which can be quiried like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00014-c891d2b5-3988-46b9-911e-ecafaa62c2b7",
    "deepnote_cell_type": "code",
    "execution_millis": 5,
    "execution_start": 1602586116848,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "nltk.help.upenn_tagset('NNPS')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00016-25101a1d-bf80-48d7-b310-1489e629caf6",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "**OBSERVE AND REFLECT** Find out what some of the other words' tags mean by replacing the tag in the previous code line. Have you come across any questionable cases? Why, in your opinion, they have got these particular tags?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00017-c65b91a8-253a-47b1-bfc7-47592f81c41f",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "**SAMPLE ANSWER**: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00019-8c9d921c-9fd2-472a-aaeb-0fc71085b379",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "\"General\" is tagged as NN (in \"secretary general\") - but general may be a noun or adjective. Here is it rather an adjective. An example with 100% noun would be \"general of the army\".\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00020-c6b276b7-ff7b-4dd4-a035-6ef99da2813e",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "\"Pandemic\" is tagged here as an adjective. Dictionary defines this word as a) adjective (meaning \"(of a disease) prevalent over a whole country or the world\" and b) noun (meaning \"an outbreak of a pandemic disease.\n",
    "\"the results may have been skewed by an influenza pandemic\"). In what meaning is this word used in the sentence? It is not an adjective because it described not a disease but policies, therefore \"pandemic policies\" is a noun phrase consisting of two nouns, where one noun modifies the other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00017-2b4125a0-3753-4812-bc4f-320c55e6a5f9",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Note how \"pandemic\" is tagged 'JJ' (adjective):  is it really an adjective here? In this particular word combination (lax pandemic policies) pandemic is a noun used in the role of an adjective (to modify another noun - policies). If you look up the word \"pandemic\" in a dictionary you will see that it is only used as an adjective when we talk of a disease (\"a pandemic disease\"). However, the tagger is based on certain rules and thinks that \"pandemic\" is an adjective. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00022-fa7f323a-cfd8-436f-aba6-00561e652d27",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "________"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00018-922b3c5e-612f-448a-8e30-21a5c8efc476",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Tagging is a **disambiguation** task (aiming at removing lexical ambiguity). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00018-6435a389-3cd8-4a04-917a-415ba4fc6490",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "**Homographs** (words that have the same spelling but more than one meaningy) present a particular challenge. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00019-1d8a9f73-8853-4c74-985e-85ad4c0cecd6",
    "deepnote_cell_type": "code",
    "execution_millis": 5,
    "execution_start": 1602166055236,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "homograph_1 = word_tokenize(\"When shot at, the dove dove into the bushes.\")\n",
    "nltk.pos_tag(homograph_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00020-7a4648cf-0351-400a-a1f5-d7b254e88d34",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "What part of speech is \"dove\" in first case and what is it in the second one? Both of them are tagged \"NN\". Problem!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00021-e8ed7b54-5ee2-419a-a31e-24ca555a34d2",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Another issue - important for speech recognition and generation - is when word meaning defines the **stressed syllable**. See below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00021-25f7366e-09c8-40ca-ad12-ba441445e479",
    "deepnote_cell_type": "code",
    "execution_millis": 41,
    "execution_start": 1602586373943,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "homonym = word_tokenize(\"They refuse to permit us to obtain the refuse permit\")\n",
    "nltk.pos_tag(homonym)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00025-95b1b682-b6eb-4850-9b7b-b7369ffd4033",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "\"Refuse\" and \"permit\" are tagged as a present tense verb (VBP) and a noun (NN). In the two cases these words have different stress positions. We need to know what part of speech the word is to pronounce it correctly. \n",
    "Similar cases: https://www.english-at-home.com/pronunciation/noun-and-verb-syllable-stress\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00026-278ac315-b39d-4450-a0be-02e4cd11a66f",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00025-f239fb0b-9440-45ef-a62c-f3a9a6a47604",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "**CODE IT** Using the nltk librabry tag the part of speeches of the followng phrases and Identify how the word **back** has been tagged in each sentence:\n",
    "\n",
    "\n",
    "pharse_1 = The **back** door (Modifier/ADJ/JJ)\n",
    "\n",
    "phrase_2 = On my **back** (NN)\n",
    "\n",
    "phrase_3 = Win the voters **back**  (RB)\n",
    " \n",
    "phrase_4 = Promise to **back** the bill (V)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cell_id": "00025-b3cdc55b-b7e9-4f09-8cae-e3274759d654",
    "deepnote_cell_type": "code",
    "execution_millis": 1,
    "execution_start": 1603806455292,
    "output_cleared": false,
    "source_hash": "a7229d4a",
    "tags": []
   },
   "outputs": [],
   "source": [
    "#insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00024-b0c7565f-8510-4a7d-af4c-05f12c015ad2",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "## Conventions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00025-473ef0d6-208f-4799-a95f-464e7c287b97",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Convention: in NLTK **a tagged token is represented by a tuple (token, tag)**:\n",
    "\n",
    "Useful function str2tuple(): \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00026-a706a8c3-333f-49a8-963a-9cd5fcdc3bc1",
    "deepnote_cell_type": "code",
    "execution_millis": 2,
    "execution_start": 1602586379610,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "tagged_token = nltk.tag.str2tuple('Göran/NNP')\n",
    "print (tagged_token)\n",
    "print (tagged_token[0])\n",
    "print (tagged_token[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00027-31acacce-dd74-4ad9-ac54-85284cfa93b0",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Some text corpora are pos-tagged, and we may need to extract a list of tagged tokens. \n",
    "1) tokenize the string to access the individual word/tag strings\n",
    "2) convert each of these into a tuple with str2tuple()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00028-436503ab-8c05-444d-af4a-d19a3e9eaafa",
    "deepnote_cell_type": "code",
    "execution_millis": 3,
    "execution_start": 1602586382077,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "sent = '''\n",
    "... The/AT grand/JJ jury/NN commented/VBD on/IN a/AT number/NN of/IN\n",
    "... other/AP topics/NNS ,/, AMONG/IN them/PPO the/AT Atlanta/NP and/CC\n",
    "... Fulton/NP-tl County/NN-tl purchasing/VBG departments/NNS which/WDT it/PPS\n",
    "... said/VBD ``/`` ARE/BER well/QL operated/VBN and/CC follow/VB generally/RB\n",
    "... accepted/VBN practices/NNS which/WDT inure/VB to/IN the/AT best/JJT\n",
    "... interest/NN of/IN both/ABX governments/NNS ''/'' ./.\n",
    "... '''\n",
    "print ([nltk.tag.str2tuple(t) for t in sent.split()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00029-dbabf313-cede-4f22-9f6c-7d37344b0751",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Brown corpus (a million words of samples from 500 written texts from different genres published in the\n",
    "WSJ United States in 1961) comes with POS tags. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00037-6712cf7f-da96-4983-a546-ac04f424a68d",
    "deepnote_cell_type": "code",
    "execution_millis": 6,
    "execution_start": 1602586430718,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "nltk.download('brown') #download Brown corpus from NLTK\n",
    "nltk.corpus.brown.tagged_words() [:40] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00031-51034f91-d9ea-43e2-99c9-8748b732e52b",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "## Practical applications of POS tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00034-07406ada-dbe0-40a1-b3b3-b39f52310a18",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "As POS tagging is a disambiguation task it is widely used in **machine translation**. \n",
    "\n",
    "- ENG: \"I fish a fish\"\n",
    "- FR (correct): \"Je pêche un poisson\". \n",
    "\n",
    "To make the two words distinct from each other from the point of view of computer, we need to POS-tag them - only then the translation will be correct. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00035-2c99521f-8e25-4711-9941-35bc72d9f063",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "** Google Translator API for python: ** [``googletrans``](https://pypi.org/project/googletrans/) library is an api for using google translator in python see an example in the following cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00035-f9e01647-073c-4ed5-97ba-8ca73b4e0696",
    "deepnote_cell_type": "code",
    "execution_millis": 2291,
    "execution_start": 1602588788091,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install googletrans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00035-f33fc0e2-8c65-4e3e-88d4-05d6f6ef7e79",
    "deepnote_cell_type": "code",
    "execution_millis": 165,
    "execution_start": 1602586513520,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from googletrans import Translator\n",
    "translator = Translator()\n",
    "\n",
    "sentence =\"I fish a fish\"\n",
    "\n",
    "translated_sentence  = translator.translate('I fish a fish', src='en',dest='fr').text\n",
    "print(translated_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00035-2151978a-bd83-4801-a9d2-9fc3e7ea2685",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "## A couple of words on how POS taggers are created [1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00036-7061376a-5898-4a5d-b0ec-ea584cdb1513",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "\n",
    "- **Lexical Based Methods** — Assigns the POS tag the most frequently occurring with a word in the training corpus.\n",
    "- **Rule-Based Methods**  — Assigns POS tags based on rules. For example, we can have a rule that says, words ending with “ed” or “ing” must be assigned to a verb. Rule-Based Techniques can be used along with Lexical Based approaches to allow POS Tagging of words that are not present in the training corpus but are there in the testing data.\n",
    "- **Probabilistic Methods** — This method assigns the POS tags based on the probability of a particular tag sequence occurring. Conditional Random Fields (CRFs) and Hidden Markov Models (HMMs) are probabilistic approaches to assign a POS Tag.\n",
    "- **Deep Learning Methods** — Recurrent Neural Networks can also be used for POS tagging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00044-a5b5d7dc-cf42-48a3-b9b0-6537694e453d",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00012-63c9d0a0-1afe-4d4d-ad21-12031025974b",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "## SpaCy : a python NLP library   [2]\n",
    "\n",
    "\n",
    "spaCy is a free, open-source library for advanced Natural Language Processing (NLP) in Python.\n",
    "\n",
    "Using Spacy:\n",
    "\n",
    "\n",
    "1. Install spacy using: !pip install spacy\n",
    "2. Choose a model from https://spacy.io/models\n",
    "3. download the model using: !python -m spacy download {model_name} \n",
    "4. Run the spacy pipeline with your model and get the tags you need.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00041-5dceec8a-ed35-429d-a9bb-231b9309d962",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "### 1. Install spaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00015-335cb694-237a-48bb-b57d-da9a2ae2cb3b",
    "deepnote_cell_type": "code",
    "execution_millis": 7932,
    "execution_start": 1602586531147,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00043-331a0c18-6971-4b19-9509-814fc0bcca68",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "### 2. Choose and download the spaCy model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00021-afac7e6b-e2c8-4d61-82b2-2a95ef35a7de",
    "deepnote_cell_type": "code",
    "execution_millis": 13002,
    "execution_start": 1603806654619,
    "output_cleared": false,
    "source_hash": "fbab500f",
    "tags": []
   },
   "outputs": [],
   "source": [
    "!python -m spacy download en_core_web_sm #downloading the English model\n",
    "\n",
    "!python -m spacy download fr_core_news_sm #downloading the French model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00028-f6fef50a-b9e6-470d-b22a-152a2d3501f6",
    "deepnote_cell_type": "code",
    "execution_millis": 1465,
    "execution_start": 1602586558092,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import en_core_web_sm\n",
    "nlp = en_core_web_sm.load()\n",
    "doc = nlp(\"Colorless green ideas sleep furiously.\")\n",
    "\n",
    "\n",
    "\n",
    "for token in doc:\n",
    "    print(token.text, token.pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00048-853e3b5d-5264-47bd-84fd-d5c47be66339",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "** Code it ** Think of a sentence in a language you speak in which one word may be tagged with two (or more) different POS tags (e.g. fish (verb) - fish (noun))  \n",
    "and translate it to another language using the Google Translator API.\n",
    "\n",
    "Then use Spacy pipeline to tag both the translation and the original sentence with POS tags. (like the: \"I fish a fish example\")\n",
    "\n",
    "** Note:  ** Try to choose a language that both Spacy and Google Translator support\n",
    "."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00052-57a21ded-51a0-4ebf-932d-ae63943e736f",
    "deepnote_cell_type": "code",
    "execution_millis": 3221,
    "execution_start": 1603808449424,
    "output_cleared": false,
    "source_hash": "914ab165",
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "from googletrans import Translator\n",
    "import en_core_web_sm\n",
    "import fr_core_news_sm\n",
    "\n",
    "\n",
    "sentence = \"I went back in trough the back door.\"\n",
    "nlp = en_core_web_sm.load()\n",
    "doc = nlp(sentence)\n",
    "\n",
    "for token in doc:\n",
    "    print(token.text, token.pos_)\n",
    "\n",
    "translator = Translator()\n",
    "\n",
    "translated_sentence  = translator.translate(sentence, src='en',dest='fr').text\n",
    "nlp_fr = fr_core_news_sm.load()\n",
    "fr_doc = nlp_fr(translated_sentence)\n",
    "\n",
    "print(translated_sentence)\n",
    "for token in fr_doc:\n",
    "    print(token.text, token.pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00033-e070903c-f8a5-4fac-938d-004a2319cec0",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "**If you fancy**  The sentence \"Colorless green ideas sleep furiously\" was introduced by Noam Chomsky as a sentence which is syntactically correct but it semantically doesn't make sense.\n",
    "\n",
    "Noam Chomsky is a highly respected contemporary linguist who is known for the controversial theory of [universal grammar](https://www.youtube.com/watch?v=517XJ3eOIzg), [generative grammar](https://www.youtube.com/watch?v=jc2bL1z9Wh4) and the minimalist program.\n",
    "\n",
    "\n",
    "Watch [this Interview in 1989](https://www.youtube.com/watch?v=hdUbIlwHRkY&t=534s) of Chomsky talking about language change.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00052-029a746d-5cc9-4543-a289-7956be55ab88",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "## Stanza : Yet another NLP Python Package  [4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00054-acb8252e-5c8a-4cd2-b7e2-7a6ea66b9146",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Stanza is a recently released python library by Stanford university for Natural Language Processing.\n",
    "\n",
    "It covers more than 60 human languages.\n",
    "\n",
    "You can run a demo of Stanza [here](http://stanza.run/)\n",
    "\n",
    "In order to use Stanza for POS tagging you need to follow the same steps as we did in using ** SpaCy **\n",
    "\n",
    "1. Download and install library using pip\n",
    "\n",
    "2. Choose and download model using stanza.download('lang') (replace lang with the abbreviation for your chosen language)\n",
    "\n",
    "3. Run Pipeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00055-0f2af887-c411-41cf-ab80-1120dde99f23",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "**NOTE:** Stanza is a huge language model the english model itself is 428MB.\n",
    "\n",
    " If you want to use stanza for another language change the language in `download` and `Pipeline` commands. You can find a list of languages supported by Stanza [here](https://stanfordnlp.github.io/stanza/available_models.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00056-c14d00bd-c05d-46ec-8b24-750f0b5805c5",
    "deepnote_cell_type": "code",
    "execution_millis": 66388,
    "execution_start": 1602586951221,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install stanza\n",
    "\n",
    "import stanza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00057-33b1ede9-80a5-40aa-8f20-28495c1ad9a6",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Using Stanza for pos-tagging a sentence in english"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00052-14217b1d-07a7-4553-b673-20c16498f125",
    "deepnote_cell_type": "code",
    "execution_millis": 1,
    "execution_start": 1603808545472,
    "output_cleared": false,
    "source_hash": "872ced7e",
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "stanza.download('en') # download English model\n",
    "\n",
    "nlp = stanza.Pipeline('en') # initialize English neural pipeline\n",
    "\n",
    "doc = nlp(\"Barack Obama was born in Hawaii.\") # run annotation over a sentence\n",
    "\n",
    "print(*[f'word: {word.text}\\tupos: {word.upos}' for sent in doc.sentences for word in sent.words], sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00059-5310be81-7973-403b-ad50-930d820f3357",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Using Stanza for pos-tagging a sentence in Hebrew"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00057-d0e4d1e5-b18c-4c74-84ff-6253ad20c71a",
    "deepnote_cell_type": "code",
    "execution_millis": 17537,
    "execution_start": 1602587204986,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "stanza.download('he') # download Hebrew model\n",
    "\n",
    "nlp = stanza.Pipeline('he') # initialize Hebrew neural pipeline\n",
    "\n",
    "doc = nlp(\"אני בקורס המעשי כרגע מנסה את החבילה החדשה הזו.\") # run annotation over a sentence\n",
    "\n",
    "print(*[f'word: {word.text}\\tupos: {word.upos}' for sent in doc.sentences for word in sent.words], sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00060-f6078cbe-d3ae-4278-aa7e-3731ef3c302a",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00062-b26be551-7f27-4e03-99fa-0ab7824237aa",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "**Homework** Create a word cloud of Proper Nouns from a text of your own choosing which is longer than 10,000 words.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00062-1df7ce18-3def-4be9-a170-513eef9f61f5",
    "deepnote_cell_type": "code",
    "execution_millis": 227516,
    "execution_start": 1603809623744,
    "output_cleared": false,
    "source_hash": "e12fe40c",
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install urllib\n",
    "import urllib\n",
    "url = \"https://www.gutenberg.org/files/98/98-0.txt\"\n",
    "file = urllib.request.urlopen(url)\n",
    "nlp = en_core_web_sm.load()\n",
    "proper_nouns=[]\n",
    "for line in file:\n",
    "    decoded_line = line. decode(\"utf-8\")\n",
    "    doc = nlp(decoded_line)\n",
    "    for token in doc:\n",
    "        if(token.pos_==\"PROPN\"):\n",
    "            proper_nouns.append(token.text)\n",
    "\n",
    "!pip install wordcloud==1.8.0\n",
    "\n",
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    " \n",
    "proper_nouns_word_cloud = WordCloud().generate(\" \".join(proper_nouns))\n",
    " \n",
    "# Display the generated image:\n",
    "fig = plt.figure()\n",
    "\n",
    "\n",
    "plt.imshow(proper_nouns_word_cloud, interpolation='bilinear')\n",
    "plt.axis(\"off\")\n",
    "plt.margins(x=5, y=5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00057-7dc40799-de02-403c-ab12-04cb8a6dec3b",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "## Refereneces\n",
    "\n",
    "**\\[1\\]** https://medium.com/analytics-vidhya/pos-tagging-using-conditional-random-fields-92077e5eaa31\n",
    "\n",
    "**\\[2\\]** https://spacy.io/\n",
    "\n",
    "**\\[3\\]** http://www.cs.columbia.edu/~kathy/NLP/2017/ClassSlides/Class5-POS/pos_F17.pdf\n",
    "\n",
    "**\\[4\\]** https://stanfordnlp.github.io/stanza/"
   ]
  }
 ],
 "metadata": {
  "deepnote_execution_queue": [],
  "deepnote_notebook_id": "b31a4866-a182-4ba4-99d7-92ad057962e2",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}