06-POS-and-Keywords-in-Context.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c0681f77-fb3f-4986-9df0-9282e3c4223f",
   "metadata": {},
   "source": [
    "# Parts of Speech and Keywords in Context"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d080c86-7c8f-4863-855b-07eda17d96bd",
   "metadata": {},
   "source": [
    "-------------------\n",
    "**Contents of this notebook**\n",
    "\n",
    "[Parts of Speech Tagging](#section-7)\n",
    "- [Finding all the adjectives in a text](#section-8)\n",
    "- [Finding all the subjects in a text](#section-9)\n",
    "- [Finding the most common adjectives associated with a given keyword](#section-10)\n",
    "\n",
    "-------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1885393f-845f-4590-b352-55d595a2eb0f",
   "metadata": {},
   "source": [
    "<a id='section-7'></a>\n",
    "## Parts of Speech Tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3707ad8d-c7cd-4e88-bbed-6bbe1cf72d2b",
   "metadata": {},
   "source": [
    "We're going to use spaCy to identify parts of speech in a text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8b0ca80-a4c6-44b3-94ad-0ec2244972e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Imports\n",
    "import spacy\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ee08643-f774-4e8f-99c1-25db266ad502",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Download the language model you're interested in\n",
    "!python -m spacy download en_core_web_md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3657dd50-5296-45e3-80fb-33c20541e674",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load language model\n",
    "nlp = spacy.load('en_core_web_md')\n",
    "\n",
    "#Create spaCy document\n",
    "text = open('soderberg-corpus/1897_Drizzle.txt', encoding='utf-8').read()\n",
    "document = nlp(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77e34b04-2ca3-43cd-b8d8-dc3e3ee94da2",
   "metadata": {},
   "source": [
    "To get part of speech tags for every word in a document, we have to iterate through all the tokens in the document and pull out the `.pos_` attribute for each token. We can get even finer-grained dependency information with the attribute `.dep_`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4aa6357-984e-4d30-a99d-372da2234b0c",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Iterate through tokens in spacy document and retrieve for each token\n",
    "#the text of that token, the POS label associated with it and the Dependency label associated with it\n",
    "for token in document:\n",
    "    print(token.text, token.pos_, token.dep_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "218d4dc8-1b60-4f34-b48a-9407f6b8ab2f",
   "metadata": {},
   "source": [
    "If you inspect the list above, you might notice it is not always completely reliable (and the quality will vary greatly for different languages). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bd46ae4-2788-43c0-8856-a3a22b7ec3b9",
   "metadata": {},
   "source": [
    "<a id='section-8'></a>\n",
    "#### Finding all the adjectives in a text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "580542f4-233a-48b8-a70a-170d1b0ec0d0",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Create an empty list\n",
    "then for loop iterating over the tokens in the document\n",
    "and append to the list if it is an adjective.\n",
    "\n",
    "You can change the parts of speech tag to whatever tag you're interested in\n",
    "e.g. adverbs (ADV), noun (NOUN), pronouns (PRON), proper nouns (PROPN), etc.)\n",
    "\"\"\"\n",
    "adjs = []\n",
    "for token in document:\n",
    "    if token.pos_ == 'ADJ':\n",
    "        adjs.append(token.text)\n",
    "adjs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d8beea2-ab6f-4ed5-8421-5db45b86d7b0",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Count the most common adjectives\n",
    "adjs_tally = Counter(adjs)\n",
    "adjs_tally.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "434e03d0-6b3d-4d4c-a038-56aa9c8f2204",
   "metadata": {},
   "source": [
    "<a id='section-9'></a>\n",
    "#### Finding all the subjects in a text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33429df9-7e0a-4d0e-b6c1-f7e9f4e72553",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Create an empty list\n",
    "then for loop iterating over the tokens in the document\n",
    "and append to the list if it is a nominal subject.\n",
    "\n",
    "You can change the dependency tag to whatever tag you're interested in\n",
    "e.g.'nsubj': nominal subjects (nsubj); direct objects (dobj); indirect objects (iobj)\n",
    "\"\"\"\n",
    "\n",
    "subjs = []\n",
    "for token in document:\n",
    "    if token.dep_ == 'nsubj':\n",
    "        subjs.append(token.text)\n",
    "subjs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28b93ebd-15b4-47ea-b3e3-a98dcd9861a1",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Count the most common subjects\n",
    "subjs_tally = Counter(subjs)\n",
    "subjs_tally.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83ff7d55-5455-4d38-9525-4b7c1a106a2d",
   "metadata": {},
   "source": [
    "<a id='section-10'></a>\n",
    "#### Finding the most common adjectives associated with a given keyword"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fab196c3-9a58-4505-ae5e-c53971a21cec",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make a list of (word) tokens and POS labels from the document \n",
    "tokens_and_labels = [(token.text, token.pos_) for token in document if token.is_alpha]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa4dd78b-7978-4b97-a3b7-6ee17582a0f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Define a function to return list of ngrams\n",
    "def make_ngrams(tokens, n):\n",
    "    ngrams = []\n",
    "    for i in range(len(tokens)-(n-1)):\n",
    "        ngrams.append(tokens[i:i+n])\n",
    "    return ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33449539-95be-40df-b2c5-deb8b9c155c9",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Call your functions\n",
    "#Change the number to change your context window\n",
    "#(i.e. how many words you want around the keyword)\n",
    "ngrams = make_ngrams(tokens_and_labels, 6)\n",
    "ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad29be33-efec-45a9-a512-334c228db716",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Define a function to return most frequent words \n",
    "#that appear next to a particular keyword\n",
    "#and are a particular parts of speech\n",
    "def get_neighbor_words_and_labels(keyword, ngrams, pos_label = None):\n",
    "    \n",
    "    neighbor_words = []\n",
    "    keyword = keyword.lower()\n",
    "    \n",
    "    for ngram in ngrams:\n",
    "        words = [word.lower() for word, label in ngram]\n",
    "        if keyword in words:\n",
    "            for word, label in ngram:\n",
    "                if label == pos_label or pos_label == None:\n",
    "                    neighbor_words.append(word.lower())\n",
    "    return Counter(neighbor_words).most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "102a44af-4451-4c09-bd3b-f09858268ecf",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Call your function\n",
    "#For example, look for most common adjectives associated with 'sun'\n",
    "get_neighbor_words_and_labels('sun', ngrams, pos_label='ADJ')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5cdcced-fe99-43fb-9ba7-54a755b01998",
   "metadata": {},
   "source": [
    "_Acknowledgements_: This notebook is inspired by Melanie Walsh’s [_Introduction to Cultural Analytics & Python_](https://melaniewalsh.github.io/Intro-Cultural-Analytics/05-Text-Analysis/Multilingual/Chinese/03-POS-Keywords-Chinese.html#keyword-extraction) and William Turkel and Adam Crymble's [\"Keywords in Context (using n-grams) with Python\"](https://programminghistorian.org/en/lessons/keywords-in-context-using-n-grams)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}