07X-POS-and-NER-in-Chinese-and-Korean.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "75dbeff0-1ee0-426c-92d0-370c1fc61432",
   "metadata": {},
   "source": [
    "## Parts of Speech Tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "542763ee-21e4-451e-b553-9ca99611da63",
   "metadata": {},
   "source": [
    "We're going to use spaCy to identify parts of speech in a text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "951c1c3e-7d43-4d67-a2b0-2847388dd03b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Imports\n",
    "import spacy\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed6febc5-4073-49e6-bbfc-5aa4bbf0dfe4",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Download the language model you're interested in\n",
    "#e.g. for Chinese: python -m spacy download zh_core_web_sm\n",
    "#e.g. for Korean: ko_core_news_sm\n",
    "#Visit: https://spacy.io/usage/models#languages for more\n",
    "!python -m spacy download ko_core_news_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e50a42cc-4035-4a49-a2cb-5152e0c891e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load language model\n",
    "nlp = spacy.load('ko_core_news_sm')\n",
    "\n",
    "#Create spaCy document\n",
    "text = open('korean-corpus.txt', encoding='utf-8').read()\n",
    "document = nlp(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6878359-80bb-4442-bdf3-b59c6e1b3031",
   "metadata": {},
   "source": [
    "To get part of speech tags for every word in a document, we have to iterate through all the tokens in the document and pull out the `.lemma_` attribute for each token, which gives us the un-inflected version of the word. We’ll also pull out the `.pos_` attribute for each token. We can get even finer-grained dependency information with the attribute `.dep_`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a85770db-65ba-4d0f-b7ea-4ed2e6131eeb",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Iterate through tokens in spacy document and retrieve for each token\n",
    "#the lemmatized version of that token, \n",
    "#the POS label associated with it and the Dependency label associated with it\n",
    "for token in document:\n",
    "    print(token.lemma_, token.pos_, token.dep_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34d9d043-d485-46bf-a85a-0f69b6585f13",
   "metadata": {},
   "source": [
    "If you inspect the list above, you might notice it is not always completely reliable (and the quality will vary greatly for different languages)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "284d850d-7fad-4533-a974-d776786e3f1d",
   "metadata": {},
   "source": [
    "#### Finding all the adjectives in a text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8866d023-ee18-41c4-aa54-8066cc8f52c9",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Create an empty list\n",
    "then for loop iterating over the tokens in the document\n",
    "and append to the list if it is an adjective.\n",
    "\n",
    "You can change the parts of speech tag to whatever tag you're interested in\n",
    "e.g. adverbs (ADV), noun (NOUN), pronouns (PRON), proper nouns (PROPN), etc.)\n",
    "\"\"\"\n",
    "adjs = []\n",
    "for token in document:\n",
    "    if token.pos_ == 'ADJ':\n",
    "        adjs.append(token.text)\n",
    "adjs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55f741aa-ebc9-4a5a-a329-eec594774ea0",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Count the most common adjectives\n",
    "adjs_tally = Counter(adjs)\n",
    "adjs_tally.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "203ded6c-30f6-4b74-a8c9-1ba1eda0983f",
   "metadata": {},
   "source": [
    "#### Finding the most common adjectives associated with a given keyword"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af9bca86-50bb-4f2e-97f8-20c4f79b66d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make a list of (word) tokens and POS labels from the document \n",
    "tokens_and_labels = [(token.lemma_, token.pos_) for token in document if token.is_alpha]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2aafa0fc-a141-4a65-8eea-b8403cfea8c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Define a function to return list of ngrams\n",
    "def make_ngrams(tokens, n):\n",
    "    ngrams = []\n",
    "    for i in range(len(tokens)-(n-1)):\n",
    "        ngrams.append(tokens[i:i+n])\n",
    "    return ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dcacd25-64f0-422e-b545-1a68e751a93b",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Call your functions\n",
    "#Change the number to change your context window\n",
    "#(i.e. how many words you want around the keyword)\n",
    "ngrams = make_ngrams(tokens_and_labels, 6)\n",
    "ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc95f650-b6fb-4177-bf0e-cf54ef0aa245",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Define a function to return most frequent words \n",
    "#that appear next to a particular keyword\n",
    "#and are a particular parts of speech\n",
    "def get_neighbor_words_and_labels(keyword, ngrams, pos_label = None):\n",
    "    \n",
    "    neighbor_words = []\n",
    "    keyword = keyword.lower()\n",
    "    \n",
    "    for ngram in ngrams:\n",
    "        words = [word.lower() for word, label in ngram]\n",
    "        if keyword in words:\n",
    "            for word, label in ngram:\n",
    "                if label == pos_label or pos_label == None:\n",
    "                    neighbor_words.append(word.lower())\n",
    "    return Counter(neighbor_words).most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0e04e91-dc5f-4582-9638-4b7945bd0fbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Call your function\n",
    "#For example, look for most common adjectives associated with 'sun'\n",
    "get_neighbor_words_and_labels('보+면', ngrams, pos_label='ADJ')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84b98349-1451-4a31-bc24-a25be2ad5e98",
   "metadata": {},
   "source": [
    "## Named Entity Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6abca9f3-69c4-4d5b-94cc-3fca904fda10",
   "metadata": {},
   "source": [
    "#### Finding all named entities in a document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "908a3c75-9383-400e-999a-710dbfcb60ee",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We can use `.ents` to pull out all the Named Entities spaCy reocgnizes in the document\n",
    "document.ents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2df8ed30-e06a-4b17-86ba-57189cd9e67c",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Get Named Entities and their label\n",
    "for named_entity in document.ents:\n",
    "    print(named_entity, named_entity.label_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5c8ebf3-cf72-4518-aa1e-6f68b36ea6c6",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Visualize all the Named Entities using displacy\n",
    "from spacy import displacy\n",
    "displacy.render(document, style=\"ent\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3e23e42-d2e0-4940-9f5a-de48b8d602bb",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Get only Named Entities of a certain type (e.g. people with PERSON)\n",
    "for named_entity in document.ents:\n",
    "    if named_entity.label_ == 'LC':\n",
    "        print(named_entity)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d216098-9353-4e95-b19c-67a80ad6dcae",
   "metadata": {},
   "source": [
    "#### Finding the most frequent Named Entities of a given type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7197166b-0d5b-4efd-bde1-25498bcb5a36",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Define a function that finds Named Entities of a given label \n",
    "def find_most_frequent_NE(doc, NE_label=None):\n",
    "    \n",
    "    named_entities = []\n",
    "    \n",
    "    for named_entity in document.ents:\n",
    "        if named_entity.label_ == NE_label or NE_label == None:\n",
    "            named_entities.append(named_entity.text)        \n",
    "    return(Counter(named_entities).most_common())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74c5bdd1-5363-4643-b3cd-70810807c523",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Call your function for a given NE (e.g. PERSON, or DATE or TIME)\n",
    "find_most_frequent_NE(document, NE_label='LC')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22d2fb44-58a9-4261-97ba-bddc6036cd14",
   "metadata": {},
   "source": [
    "_Acknowledgements_: This notebook is inspired by Melanie Walsh’s [_Introduction to Cultural Analytics & Python_](https://melaniewalsh.github.io/Intro-Cultural-Analytics/05-Text-Analysis/Multilingual/Chinese/03-POS-Keywords-Chinese.html#keyword-extraction) and William Turkel and Adam Crymble's [\"Keywords in Context (using n-grams) with Python\"](https://programminghistorian.org/en/lessons/keywords-in-context-using-n-grams)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}