00-Introduction_Jupyter-Notebooks-Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c5ad3708-ed6d-4ffd-83e8-0865e517a610",
   "metadata": {
    "tags": []
   },
   "source": [
    "# JupyterHub\n",
    "\n",
    "### Accessing JupyterHub\n",
    "\n",
    "- Got to https://jhub.dartmouth.edu/  \n",
    "- Log in using your university credentials  \n",
    "- Click \"start my server\"  \n",
    "- Select \"COLT 70 -- Fall 2023\"  \n",
    "- It may take some time for the server to start up   \n",
    "- Notebooks are listed on left (with file extension \".ipynb\")\n",
    "\n",
    "### Editing a Notebook\n",
    "\n",
    "At this point you can only view a notebook (by double clicking on it). In order to be able to edit the notebook copy-paste the notebook into your home directory:  \n",
    " - right click on the notebook file you want to edit (listed on the left)  \n",
    " - select \"copy\"  \n",
    " \n",
    " ![right-click](right-click.png)  \n",
    " \n",
    " \n",
    " - double click on your home directory (by clicking the directory icon above the list of notebooks)  \n",
    " \n",
    " ![home-directory](home-directory.png)  \n",
    " \n",
    " - paste the notebook  \n",
    "\n",
    "You should now be able to edit and run the cells of the notebook.\n",
    "For the Introduction to Python below, also copy over the \"gilman.txt\" file and the gilman-corpus directory. \n",
    "\n",
    "N.B. you can check if a notebook is in read-only or in editable (writable) mode by hovering your cursor over the name of the notebook file — this will list some information about the notebook (such as its name, size, the path to the notebook (i.e. where the notebook is located), its date of creation and the last time modified, and the type of kernel it is running on). If it lists “true” next to “Writable” then the notebook is editable (in _writable_ mode). \n",
    "\n",
    "### Uploading files\n",
    "\n",
    "To upload files to JupyterHub:  \n",
    " - make sure you are in your home directory (where you paste the writable/editable notebooks)  \n",
    " - click the upload icon  \n",
    " \n",
    " ![uploadscreenshot](uploadscreenshot.png)  \n",
    " \n",
    " - select your files to upload"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba2b6262-df90-4dfb-b40a-d355072303e5",
   "metadata": {},
   "source": [
    "-------------------\n",
    "# Contents of this notebook  \n",
    "\n",
    "[Introduction to Notebooks](#section-1)\n",
    "- [Editing and running code in notebooks](#section-2)   \n",
    "\n",
    "[Introduction to Python](#section-4)\n",
    "- [Imports](#section-5)\n",
    "- [Functions](#section-6)\n",
    "- [Variables](#section-7)\n",
    "- [Data types](#section-8)\n",
    "- [Methods](#section-9)\n",
    "- - [List methods](#section-10)\n",
    "- - [String methods](#section-11)\n",
    "- [For loops](#section-12)\n",
    "- [Conditionals and Boolean values](#section-13)\n",
    "- [Flow of execution and Comments](#section-14)\n",
    "- - - [Clearing outputs: interrupting and restarting the kernel](#section-3) \n",
    "- [Working with text files](#section-15)\n",
    "- - [Opening and reading text files](#section-16)\n",
    "- - [File paths](#section-17)\n",
    "- - [Working with multiple files](#section-18)\n",
    "- [Wrap up](#section-19)\n",
    "- [Debugging](#section-20)\n",
    "-------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04f42d94-a782-4265-a0f8-45e81bd97d4b",
   "metadata": {},
   "source": [
    "<a id='section-1'></a>\n",
    "# Introduction to Notebooks  \n",
    "\n",
    "This is a notebook! A notebook is composed of different cells which can be used to write markdown text or computer code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9016642f-0731-45ad-8412-8117d27892d8",
   "metadata": {},
   "source": [
    "If you double click on this text, you will see that this is a \"Markdown\" cell for writing text, inserting images, videos, links etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b29bfc7-c936-4179-a58d-11b70ec69623",
   "metadata": {},
   "outputs": [],
   "source": [
    "## This is a code cell"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6bbb6aa-3ae6-4093-840a-fc1370ebf087",
   "metadata": {},
   "source": [
    "N.B. You can select the cell type at the top of this document with the drop down menu. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "561b8f76-a842-4f20-a0d8-90cf65f94576",
   "metadata": {},
   "source": [
    "Jupyter notebooks therefore allow you to combine code, text, images, visualizations etc. all in one place. You can edit and run code in a notebook which makes it an ideal place to play around with and test out code. The possibilities of combining code and text afforded by Notebooks are not only useful as a pedagogical and learning environment, but can also offer a way to make our analytical process explicit and to reflect on our analytical process:\n",
    "\n",
    "> \"Notebooks are theory — not merely code as theory but theory as thoughtful engagement with the theoretical work and implications of the code itself. Disciplinary norms— including contextual framing, theory, and self or auto-critique— need to accompany, supplement, and inform any computational criticism. Revealing as much of the code, data, and methods as possible is essential to enable the ongoing disciplinary conversation. Compiling these together in a single object, one that can be exported, shared, examined, and executed by others, produces a dynamic type of theorization that is modular yet tightly bound up with its object.\" (Dobson, James E. _Critical Digital Humanities: The Search for a Methodology_. Urbana-Champaign: University of Illinois Press (2019) p. 40)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67c3f65b-5977-4008-a96d-676dfcde9879",
   "metadata": {},
   "source": [
    "<a id='section-2'></a>\n",
    "### Editing and running code in notebooks\n",
    "\n",
    "To edit a cell, double click the cell. \n",
    "\n",
    "To run a code cell (or render a Markdown cell):  \n",
    "> - select the cell (a blue line will appear on the left when the cell is selected)  \n",
    "> - click the \"play\" icon at the top of the notebook OR press shift+enter.\n",
    "\n",
    "Note that the code cells have a pair of square brackets with a colon next to them [ ]:  \n",
    "Once you run a code cell a number will appear on the left. This tells you how many times the cell was run and in what order — this can help you keep track of which cells were run and in what order.  \n",
    "\n",
    "Try running the code cell above a couple of times and see the number in square brackets change.  \n",
    "\n",
    "Now run the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ecb82ec-8731-4a49-87fe-835a4488c05a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Waiting 5 seconds...')\n",
    "import time\n",
    "time.sleep(5)\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6961538-050d-4536-9a54-e28ac7f5d25a",
   "metadata": {},
   "source": [
    "Did you notice the asterisk in the brackets when running the cell? An asterisk displays whilst code is still busy executing, and the number appears when it has finished executing.  \n",
    "\n",
    "Beneath the code cell, any output from running the code will appear. In the case of the cell above \"Waiting 5 second...\" and \"Done\" were printed beneath the cell."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "891a82a1-2a84-40a2-aa02-a8cc6f05ad94",
   "metadata": {},
   "source": [
    "<a id='section-4'></a>\n",
    "# Introduction to Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d96cd9e-0841-43ec-858a-33034b20c2a8",
   "metadata": {},
   "source": [
    "Writing code involves describing a series of steps (detailed instructions) to go through in order to perform a task. Writing code involves _expressing_ instructions that you want to realize in programmatic language, the instructions you have provided are _evaluated_ which results in an _output_. Code allows you to specify instructions that the computer will follow. Whilst programming languages are in some ways very rigid — the *exact* syntax and instructions need to be defined in order for it to work — this doesn’t mean there isn’t some degree of flexibility. There are multiple ways of achieving something through code, and people have different coding styles that might differ according to their goals (e.g. is the code efficient? is the code readable? etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e67be36-4b4a-49da-a5a9-24224468a960",
   "metadata": {},
   "source": [
    "### Anatomy of a Python Script"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6837dfd-d7d1-4dd7-9cc4-b588fe667600",
   "metadata": {},
   "source": [
    "Here is an example of a programme. The end-goal (output) of this programme is to count and display the most frequent words in a given text. We're going to go through this example script and break it down to look at some basics of the Python programming language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86d08b18-9d53-400f-9635-bcd0903e18b5",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Importing the libraries and modules we'll need\n",
    "import re\n",
    "from collections import Counter\n",
    "\n",
    "#Defining a function to split our text into words\n",
    "def split_into_words(any_chunk_of_text):\n",
    "    lowercase_text = any_chunk_of_text.lower()\n",
    "    split_words = re.split(\"\\W+\", lowercase_text)\n",
    "    return split_words\n",
    "\n",
    "#Defining variables we're going to need\n",
    "#Giving the path to the text we're going to analyze\n",
    "filepath_of_text = \"gilman.txt\"\n",
    "\n",
    "number_of_desired_words = 40\n",
    "\n",
    "#Defining a list of stopwords\n",
    "stopwords = ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours',\n",
    " 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers',\n",
    " 'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves',\n",
    " 'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are',\n",
    " 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does',\n",
    " 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until',\n",
    " 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into',\n",
    " 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down',\n",
    " 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here',\n",
    " 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',\n",
    " 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so',\n",
    " 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', 'should', 'now', 've', 'll', 'amp']\n",
    "\n",
    "#Opening and Reading the text to analyze it\n",
    "full_text = open(filepath_of_text, encoding=\"utf-8\").read()\n",
    "\n",
    "#Splitting the text into words by using our previously defined function\n",
    "all_the_words = split_into_words(full_text)\n",
    "\n",
    "#Making a list of words with our stopwords removed\n",
    "meaningful_words = []\n",
    "for word in all_the_words:\n",
    "        if word not in stopwords:\n",
    "            meaningful_words.append(word)\n",
    "\n",
    "\"\"\" Another way of writing this for loop\n",
    "meaningful_words = [word for word in all_the_words if word not in stopwords] \"\"\"\n",
    "\n",
    "#Counting how many times each word in our \"meaningful_words\" list appears\n",
    "meaningful_words_tally = Counter(meaningful_words)\n",
    "\n",
    "#Pulling out the top 40 most frequently occurring words from our complete tally using hte \"most_common\" method\n",
    "most_frequent_meaningful_words = meaningful_words_tally.most_common(number_of_desired_words)\n",
    "\n",
    "#Display these top 40 most frequent words\n",
    "most_frequent_meaningful_words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "562516a1-8b89-4287-9525-1bc3a7bc1502",
   "metadata": {},
   "source": [
    "If you're encountering a `FileNotFound` error when you run the program, check that you have copy and pasted the \"gilman.txt\" file from the notebooks repository into your home directory (just as you did with the notebook). The \"gileman.txt\" file and the notebook need to be in the same place (more on this in the Filepaths section below in Working with text files)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83620919-69ea-462d-b663-8330d006b222",
   "metadata": {},
   "source": [
    "<a id='section-5'></a>\n",
    "#### Imports\n",
    "\n",
    "There is a lot of code that has already been written, and writing your own code can mostly involve using and adapting code already written by others to your own needs.  \n",
    "\n",
    "At the top of a program, it is common to import any libraries or modules you might need later on in the program.  \n",
    "\n",
    "Python incudes many libraries (importable packages of already written code) that can do different things. In this case we are importing the library `re` (for regular expressions, which will allow us to use regular expressions to clean up our text) and from the `collections` library we’re importing the module `Counter` (which will allow us to count things)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "619512a9-95a4-461a-9f89-faeb91342a82",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "844a744d-751e-40d8-8cbd-6b7e6ba136a5",
   "metadata": {},
   "source": [
    "As a fun anecdote: There is a module in Python (called `this`) that prints the “Zen of Python” (a set of principles that informs the Python philosophy written by one of Python's developers) when you import it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32cd1944-3673-4a93-9946-5187d6948d12",
   "metadata": {},
   "outputs": [],
   "source": [
    "import this"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6c6e2ab-2221-4b11-b71d-3342a9769299",
   "metadata": {},
   "source": [
    "<a id='section-6'></a>\n",
    "#### Functions\n",
    "\n",
    "Functions are bundles of code that performs a particular task. Often a function involves giving some kind of input or specifications to the function (called “arguments” that you pass to the function in brackets), the function then performs its task on the input and returns an output (the result of its task on this particular input).\n",
    "\n",
    "Python has a number of prewritten (or built-in) functions. For example, the function `print()` will print to the screen the argument you pass to it.  \n",
    "\n",
    "Run the cell below. And then try adding your name to “Hello!” and run the cell again. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43c93846-31b1-4a37-9549-d7a37cc9c4f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Hello!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eae5e8e-9268-4471-a8a6-2b84f624e459",
   "metadata": {},
   "source": [
    "Functions performs different tasks. Notice how the same input with a different function returns a different results (can you guess what this `len()` function does from the returned output?):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da4e404d-089f-4a43-bd2b-f8c8cc4c5661",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(\"Hello!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54654c82-406f-4b66-b9fe-d24045a060fa",
   "metadata": {},
   "source": [
    "**_Defining Functions_**\n",
    "\n",
    "You can also write your own functions. This is called defining a function. \n",
    "\n",
    "It is useful to write your own functions if you want to bundle together a series of steps to achieve a particular goal. This make the code neater and easier to manage. It also means you can re-use that function anywhere in the code, you just have to “call” it by its name. \n",
    "\n",
    "In this case we have defined a function called `split_into_words` that will split our text into individuals words and that takes `any_chunk_of_text` as its argument.  \n",
    "- To define a function we write a function definition statement: starting with the keyword `def` (short for define) then the name we want to give the function and in brackets any parameters we might need to pass to the function (or nothing in brackets if no parameters are necessary). All this is followed by a colon `:`.  \n",
    "- Then there is an indented block of code that specifies what the function does. Note that this block of code is indented. Indentation is important in python as it defines where the block of code starts and ends. Anything indented will be part of the function definition. If it is not indented it will be considered not part of the function definition.  \n",
    "- The`return` statement at the end of the function sends the value that results from the function execution out to where the function was called from.  \n",
    "\n",
    "We will go into the details of what the function `split_into_words` is doing further below in the Methods section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dad04697-fc3b-4cd5-8d46-95cd3ab2f271",
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_into_words(any_chunk_of_text):\n",
    "    lowercase_text = any_chunk_of_text.lower()\n",
    "    split_words = re.split(\"\\W+\", lowercase_text)\n",
    "    return split_words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "082b4366-eab4-4e57-9768-818da6f8f061",
   "metadata": {},
   "source": [
    "<a id='section-7'></a>\n",
    "#### Variables\n",
    "\n",
    "Next in our program we see we have defined some “variables”. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37abaa11-98cd-4d2a-94a3-c1e611f934ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "filepath_of_text = \"gilman.txt\"\n",
    "\n",
    "number_of_desired_words = 40"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "781c74df-9eb1-4aee-a10c-2d0af218c958",
   "metadata": {},
   "source": [
    "Variables are like containers that store information. You assign to a variable a value that you want to use later on so that you only need to use the name of the variable later on. The value of a variable can be overwritten if you assign a new value to the same name. \n",
    "\n",
    "You can give a variable almost any name you want, but there are a few limitations. The name can’t start with a number (it needs to start with a letter), there can be no spaces in the name and no punctuation apart from underscore (\\_\\), and it cannot be a python reserve word (i.e. a word that already has a specific meaning in the python language). \n",
    "\n",
    "For example here we assign the value `“Hello!”`to a variable called greeting. And then we can pass that value to a function such as print instead of writing out “Hello!” again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ee54f52-547c-4847-b721-2515d43b5b27",
   "metadata": {},
   "outputs": [],
   "source": [
    "greeting = \"Hello!\"\n",
    "print(greeting)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96a7d7a1-5973-410f-a350-93319e5f90b4",
   "metadata": {},
   "source": [
    "<a id='section-8'></a>\n",
    "#### Data Types\n",
    "\n",
    "Notice how the variables we have assigned in our program look a bit different. One is a number that appears in green. The other are some words that appear in red in between quotation marks. \n",
    "\n",
    "This is because numbers and words are different data types in Python: depending on what type of data something is there are different rules about how to write them into code, and different possibilities about what you can do with them and how you can use them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd1a8c95-15e1-47ca-a68f-46cb40901e6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "filepath_to_text = \"gilman.txt\"\n",
    "\n",
    "number_of_desired_words = 40"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0965092f-8f00-46fa-81f2-d12397fff813",
   "metadata": {},
   "source": [
    "The function `type()` returns what type of data something is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f0bf3ea-7899-4324-9736-0b45401a4858",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(\"Hello!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a2583ec-9675-485e-a37c-fe2f2df64538",
   "metadata": {},
   "source": [
    "Let’s start by looking at four basic data types: \n",
    "\n",
    "*String*: strings store characters such as letters or numbers. String are essentially text. To specify something is a string you need to write it in “quotation marks” (both ‘single’ or “double” quotation marks work).\n",
    "\n",
    "Try finding out what data type these examples are below using the `type()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ff95b90-6c83-4048-91fe-797ca690b5c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "example = 'forty'\n",
    "another_example = \"40\"\n",
    "a_final_example = 40\n",
    "type(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e28ffe35-8bc5-49f8-9704-9926e9d9a63d",
   "metadata": {},
   "source": [
    "As you can see, the final example is a not a string (or text), it was not written in quotation marks. It is an *integer* (i.e. a whole number). Decimal numbers in python are called *floats* (or floating-point number). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3a89151-313f-4b62-86f8-1e2d6be75e00",
   "metadata": {},
   "outputs": [],
   "source": [
    "integer_example = type(40)\n",
    "float_example = type(4.5)\n",
    "print(integer_example)\n",
    "print(float_example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b9cd89f-b9f8-424c-a59a-c24b269e952d",
   "metadata": {},
   "source": [
    "There are different things you can do with different data types. See here how the exact same syntax produces different outputs depending on whether the data type is a string or an integer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8bd53d1-e941-4650-beca-85b111ee574b",
   "metadata": {},
   "outputs": [],
   "source": [
    "addition = a_final_example + a_final_example\n",
    "addition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3b0e3f5-d8da-4292-a29a-d9efcef116ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "addition = another_example + another_example\n",
    "addition"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6ce652a-6f76-47a9-bc59-94828dbbd957",
   "metadata": {},
   "source": [
    "We’ve also assigned another kind of data type to the variable “stopwords”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18164612-9428-4af7-9759-b66969431cca",
   "metadata": {},
   "outputs": [],
   "source": [
    "stopwords = ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours',\n",
    " 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers',\n",
    " 'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves',\n",
    " 'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are',\n",
    " 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does',\n",
    " 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until',\n",
    " 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into',\n",
    " 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down',\n",
    " 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here',\n",
    " 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',\n",
    " 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so',\n",
    " 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', 'should', 'now', 've', 'll', 'amp']\n",
    "\n",
    "type(stopwords)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a387adf7-73e8-48f2-a548-c0f0b1df8ade",
   "metadata": {},
   "source": [
    "The `type()` function tells us this is a _list_. A list is one of Python’s collection types of data types. Collection types are data types that allow us to store many values together in a single variable. Differences between collection types involve how items are stored and how we can retrieve or access them. \n",
    "\n",
    "With a list you can list a collection of items (between square brackets and separated by commas) into a single list. Here we are listing a collection of strings and integers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8e5fd68-a770-41d2-bdad-eafca0acf970",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_items = [4, \"apples\", 1, \"pear\", 5, \"oranges\", 6, \"melons\", 23, \"kiwis\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae365020-0065-4a19-95f1-c95528771dc2",
   "metadata": {},
   "source": [
    "Another kind of collections type in Python is a _dictionary_. In a dictionary you can list a collection of items (between curly brackets and separated by commas) with particular values assigned to each of those items. If we were to rewrite our previous list as a dictionary we could write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59a70a44-929c-4247-acc3-babce2a7dc27",
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary_items = {\"apples\": 4, \"pear\": 1, \"oranges\": 5, \"melons\": 6, \"kiwis\": 23}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03c3879f-70bb-440a-9a55-7420de1d1b14",
   "metadata": {},
   "source": [
    "If we wanted to retrieve an item from a list, we can do this using its index (or location in the sequence of items in the list)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "391f6619-311e-4f4a-90f0-28ef65a8d6f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_items[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a3805f4-be7f-413e-8ce2-5a0e011556ad",
   "metadata": {},
   "source": [
    "Note how when we retrieve the item at index “1” in the list it gives us the second item. That is because Python counts from 0, so the first item in the list is at index \\[0]\\. Try retrieving different items from the list using different index numbers. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43e4ffcf-3c60-4acf-abc4-f99335331cf7",
   "metadata": {},
   "source": [
    "You can also use reverse index to get the last item on the list without knowing how many items are in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "981ca044-dcbd-4349-8f6c-d88205f0a619",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_items[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "521baaa0-7442-4b80-b0a4-ef7e8952aedd",
   "metadata": {},
   "source": [
    "You can also take a slice of items in your list by using index numbers and colons: "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a334b1-1182-4a14-9782-184ba08d4a63",
   "metadata": {},
   "source": [
    "- You can slice from the start of the list (i.e. index [0]) up to a given index (excluding index given):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42dda058-27c6-4a79-a00f-50212130aade",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_items[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1f4f4b6-e758-4cc1-a0cf-2cfe8471afe1",
   "metadata": {},
   "source": [
    "- You can slice from the end of the list until a given index: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "488ac5eb-8130-4932-b9eb-97ab85e1e618",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_items[2:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1ee480c-3684-43c4-8c97-7afa0e2028d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_items[-2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83c04716-ce47-40a3-9d22-e5ed039acf7f",
   "metadata": {},
   "source": [
    "In a dictionary, items are not ordered according to their sequence, like in a list. They are ordered as _key-value_ pairs. Each item (key) has a value associated to it. We can therefore retrieve values by using their associated key, rather than using the index number in the sequence (as we did in a list). You can also use reverse index to get the last item on the list without knowing how many items are in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eaf267be-ea16-49cb-ba18-df78388789ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary_items[\"apples\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5560ee84-2114-48ca-811e-3b31b9db1ce6",
   "metadata": {},
   "source": [
    "<a id='section-9'></a>\n",
    "#### Methods \n",
    "\n",
    "Different data types therefore allow us to structure different kinds of information in different ways, and allow us to do different things with the information. There are things that you can only do with strings or with lists. The way you interact and do things with data types is through _methods_."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a118194-d70f-4665-b74c-f448e68d7ab2",
   "metadata": {},
   "source": [
    "<a id='section-10'></a>\n",
    "#### List methods\n",
    "\n",
    "The syntax for writing methods is an input (i.e. what you’re applying the method to) followed by a dot and the name of the method and brackets (in which we can pass the method any arguments, any particular specifications, just like we could with a function). \n",
    "\n",
    "For example, there are list methods for adding items to a list, removing items from a list, or sorting items in a list."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d48e1b33-5096-4312-845e-cd40513daaf8",
   "metadata": {},
   "source": [
    "- Add an item to a list with the `.append()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44395c82-a598-4185-8b61-32ba1c24a199",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_items = [\"apples\", \"pear\", \"oranges\", \"melons\", \"kiwis\"]\n",
    "list_items.append(\"blueberries\")\n",
    "list_items"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd5821ee-9050-4417-8597-5dfb9d29c702",
   "metadata": {},
   "source": [
    "- Remove the first instance of an items with the `remove()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8047a91d-0ea5-4213-8ffa-057d6d7bcce5",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_items.remove(\"blueberries\")\n",
    "list_items"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a1ea9de-a871-4174-8467-89aacb243184",
   "metadata": {},
   "source": [
    "- Sort the order in which the items in a list appear with the `sort()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95d40ea1-cae7-4c3b-b325-0908592167c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_items.sort()\n",
    "list_items"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2babf774-c70c-4b98-8b86-91879519601c",
   "metadata": {},
   "source": [
    "The default is for items to be sorted in ascending order (from small to large number or in alphabetical order).  \n",
    "`list.sort(reverse=False)`  \n",
    "You can change the default by setting the default parameter to `True` to sort in descending or reverse order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac99fea9-0fb6-4ab6-b211-83cea4945c64",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_items.sort(reverse=True)\n",
    "list_items"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c419920-3bdb-42ac-b018-d78dc9b45c2b",
   "metadata": {},
   "source": [
    "- If you want to add multiple items you could extend the list by adding another list to the list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e591599a-fb07-4a34-941d-1b1b2fd6657d",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_items = [\"blueberries\", \"grapes\"]\n",
    "list_items.extend(new_items)\n",
    "list_items"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eef91b59-be66-40e8-819f-9203a0a1c1e4",
   "metadata": {},
   "source": [
    "<a id='section-11'></a>\n",
    "#### String Methods\n",
    "String methods provide different ways of interacting with and doing things with strings (text). \n",
    "\n",
    "For example, you could turn all characters in a string sequence to upper case or lower case using the `upper()` and `lower()`string methods. (Why this is important will become clearer later when we talk about pre-processing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99516ac2-7c34-4ca6-b8ee-2c5b088ba52b",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_string = \"Here is an example.\"\n",
    "\n",
    "example_string = example_string.upper()\n",
    "print(example_string)\n",
    "\n",
    "example_string = example_string.lower()\n",
    "print(example_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1274c05-43e0-4761-af34-0b8904862511",
   "metadata": {},
   "source": [
    "Note how the variable `example_string` gets rewritten with new content. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6358e62-ab8e-4301-923e-f382104e6d40",
   "metadata": {},
   "source": [
    "Another useful string method to know about is `replace()`. This method replaces a defined string with a new string. For example, sometimes there are encoding issues when scrapping text and we might have our text look like this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b14115df-da8f-446a-9ceb-9010b715c833",
   "metadata": {},
   "outputs": [],
   "source": [
    "replace_example = \"Here\\xa0is\\xa0an\\xa0example.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "016f1495-4d89-48b0-bc96-3a5686e2a0ba",
   "metadata": {},
   "source": [
    "“\\xa0” is the Unicode code-point for a space. We would probably want the replace the unicode code-point with a space and re-encode our text. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23964192-97b2-461f-b617-a592033bc97c",
   "metadata": {},
   "source": [
    "What's a Unicode code-point?  \n",
    "Encodings are systems that define relations between two systems of representation for example, between numbers and letters or characters. Unicode is a standard that aspires to define a unique ID number (or “code-point\") for every character in all kinds of different languages around the world (and even emojis). This unique code-point can then be used to encode words into bits in order to represent text digitally (through the unicode encodings such as utf-8)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30b63b1b-6249-40f5-91a4-38e3978f672f",
   "metadata": {},
   "outputs": [],
   "source": [
    "replace_example = replace_example.replace(\"\\xa0\", \" \").encode(\"utf-8\")\n",
    "replace_example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35b3503-241d-4b94-aa03-8bb7a6c53487",
   "metadata": {},
   "source": [
    "In our example program for counting words, we used string methods to lower case our text and to split the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "260e832a-1272-4a85-82aa-15d0a4f5f573",
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_into_words(any_chunk_of_text):\n",
    "    lowercase_text = any_chunk_of_text.lower()\n",
    "    split_words = re.split(\"\\W+\", lowercase_text)\n",
    "    return split_words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "566e1a86-9926-46ad-9970-8d277dde84b3",
   "metadata": {},
   "source": [
    "The `split()` method is a string methods that splits any string it is given according to a specified pattern or “delimiter”. It returns a list of substrings that have been split according to the specified delimiter. The default delimiter (i.e. if we were to leave the brackets blank) is a space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59b9bcc2-eed2-4c8b-acec-5477a8d2d650",
   "metadata": {},
   "outputs": [],
   "source": [
    "split_example = \"I’m wondering how different split methods going to deal with-and split-text in different ways.\"\n",
    "split_example = split_example.split()\n",
    "split_example "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c05b7fe0-a597-4f76-b717-00c1404893cf",
   "metadata": {},
   "source": [
    "In our example program, we use a _regular expression_ to specify a particular pattern or delimiter according to which our text must be split. Regular expressions (or regex) are sequences of patterns or symbols that specify patterns in text. They’re like a code for referring to particular patterns in language in order to make searching for those patterns more concise and precise. In the case of our program for example, we can use a regular expression to specify that we want to split our text every time we encounter a character that is NOT a word character (letter). This is specified with the regular expression `\\W+`.  \n",
    "\n",
    "Compare the output of splitting with the regular expression compared to our previous method of using the default parameter of `split()`. Which would you prefer to use in your program?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83b2c1af-52a1-4077-b6e7-536765536010",
   "metadata": {},
   "outputs": [],
   "source": [
    "split_example = \"I’m wondering how different split methods going to deal with-and split-text in different ways.\"\n",
    "split_example = re.split(\"\\W+\", split_example)\n",
    "split_example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c06ab2b-1ccb-4cf4-a21d-9f991f72ba67",
   "metadata": {},
   "source": [
    "<a id='section-12'></a>\n",
    "#### For Loops\n",
    "\n",
    "Computers aren’t good at manipulating so-called \"unstructured data\" such as text. In order to be able to manipulate text programmatically, we often need to restructure it in a way that makes it manipulable, such as in a list, as above, when we split our text into a list of words. \n",
    "\n",
    "Now that the text is structured as a list, we can prgorammatically manipulate it in different ways. Often, we will want to do something with every item in the list. To do this, we will need to iterate through every item in our list by using a `for` loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cb8ae1b-cd31-49e9-94dd-4e7426bd2f76",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_text = \"It was the best of times, it was the worst of times.\"\n",
    "tokenized = sample_text.split()\n",
    "\n",
    "for word in tokenized:\n",
    "    print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8de02003-29e5-42fa-947b-69a5640bd727",
   "metadata": {},
   "source": [
    "To construct a `for` loop: you use the keyword `for` then a variable name that stands in for each item (such as here we used to word `word` but we could’ve used `item` or even `potato`), then you use the keyword `in` and the name of the list you are looping through followed by a colon `:`. Just like with a function, the indented body of code after the colon specifies what task is to be performed on every item in the list. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76bb79ca-7e0a-4cf7-b77e-b19480d46d1c",
   "metadata": {},
   "source": [
    "<a id='section-13'></a>\n",
    "#### Conditionals and Boolean values\n",
    "\n",
    "In our program example, we use a `for` loop to remove stopwords from our text."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f5e9d79-99c3-41dc-aac3-3df77c8c35ea",
   "metadata": {},
   "source": [
    "What are stopwords?  \n",
    "We will talk about this more later in the course, but when we’re analyzing text computationally we are often relying on counting words and on word frequencies. However, some very frequent words might not be very significant or relevant for our analyses. In this case, we need to remove these words from our analyses. These are called stopwords, words we remove from our analyses."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de662778-f58d-47b0-9221-06ca35379ed3",
   "metadata": {},
   "source": [
    "In order to remove the stopwords from our text (which we have restructured as a list through our `split_into_words(any_chunk_of_text)` function using the `split` method), we use a `for` loop to iterate over every item in our list and consider whether it is included in the stopwords list or not. If it is not included in the stopword list, we append it to a new list which will contain all words from our text that are not in the stopword list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbbe8e1b-9ad4-4131-a1cf-7ea748398984",
   "metadata": {},
   "outputs": [],
   "source": [
    "meaningful_words = []\n",
    "for word in all_the_words:\n",
    "        if word not in stopwords:\n",
    "            meaningful_words.append(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e010aa07-57a7-44fa-9f63-2cb098c49f19",
   "metadata": {},
   "source": [
    "We do this process of assessing whether a word in the stopword list or not by using a _conditional statement_. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1945b81c-3502-4103-855d-4229ddf797ba",
   "metadata": {},
   "source": [
    "Conditional statements (or `if` statements) follow a similar syntax to that of the `for` loop (i.e. with keyword, colon and indented block of code that performs a particular task).  \n",
    "\n",
    "A conditional statement defines a condition and specifies an action to be taken depending on the state of the condition. In the case of our program example above, if a word is not in the stopword list, then it is appended to the new list `meaningful_words`. Conditional statements therefore allow us to change the program's behavior based on whether some condition is met or not. \n",
    "\n",
    "To define different kinds of conditions we can use comparison operators such as: \n",
    "- equal to: `==`\n",
    "- not equal to: `!=`\n",
    "- greater than: `x > y`\n",
    "- lesser than: `x < y`\n",
    "- greater than or equal to: `x >= y`\n",
    "- lesser than or equal to: `x <= y`\n",
    "- object identity: `x is y`\n",
    "- negated object identity: `x is not y`\n",
    "\n",
    "If the condition is met (or `True`) then one course of action is followed, and if the condition is not met (or `False`) then another course of action is followed. These evaluations of `True` and `False` are called _Boolean values_ and are another kind of data type in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "301b781c-daa5-434d-ba10-85233759f153",
   "metadata": {},
   "source": [
    "For example, we could write a `for` loop that iterates through a list of words and evaluates whether each word is longer than two characters. If this is `True`, then the word is printed, if it is not true (`False`) then it is not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ab88f6c-7ead-4be1-8ca2-3d1049975ff1",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_text = \"It was the best of times, it was the worst of times.\"\n",
    "tokenized = sample_text.split()\n",
    "\n",
    "for word in tokenized:\n",
    "    if len(word) > 2:\n",
    "        print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "667cc464-2311-46d2-a5e6-18aa29b7e16f",
   "metadata": {},
   "source": [
    "<a id='section-14'></a>\n",
    "#### Flow of execution and Comments\n",
    "\n",
    "Python code runs from left to right and from top to bottom, one line at a time. For code to work, things need to happen in a certain order. Statements such as conditional statements can control this flow of execution, i.e. the order in which code runs. They can define whether a program follows one path of action or another.\n",
    "\n",
    "This idea of a flow of execution might be a little counter-intuitive in Notebooks because Notebooks allow you to run cells in any order, but if you start by running a cell from the bottom of a notebook that relies on a previous cell then you will probably get an error (or if you forget to run a cell then you might get an error in a subsequent cell). The output of any cell that you run in a Notebook will be carried over to the next, but you need to have run the cell and in the right order. If you get errors, check that you have run the previous cells by checking the numbers in \\[brackets\\] next to the code cells."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "384871e3-a55c-41ec-874d-04e851df6d53",
   "metadata": {},
   "source": [
    "<a id='section-3'></a>\n",
    "### Clearing outputs: interrupting and restarting the kernel\n",
    "\n",
    "If your code seems to be getting stuck, or if the flow of execution of the cells has become mixed up, it’s a good idea to make a fresh start and clear all the outputs. \n",
    "\n",
    "To clear the outputs of a single cell:  \n",
    "> select the cell > click Edit tab > select Clear Output\n",
    "\n",
    "> OR right click the cell and select Clear Output\n",
    "\n",
    "To clear the output from all cells:  \n",
    "> Edit tab > Clear All Outputs\n",
    "\n",
    "> OR right click the cell and select Clear All Outputs\n",
    "\n",
    "Interrupt and restart the kernel:  \n",
    "> You can click the “stop” and “restart” icons at the top of the notebook (next to the “play” icon)\n",
    "or  \n",
    "> Select Kernel tab and select the options you want (Interrupt Kernel, Restart Kernel).  \n",
    "\n",
    "Clearing outputs clears all outputs from executed code. Restarting the kernel clears the outputs and fires up the backend component that actually runs the code written in the Notebook. Notebooks are browser-based documents that run through your browser. Every time you open up a notebook it runs on a local server on your personal computer (that’s why it sometimes takes some time to load and why you can edit a notebook without altering the “original” notebook). If you restart the kernel it refreshes that environment which runs your Notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f0d112a-0aff-4449-be7e-f517727320ab",
   "metadata": {},
   "source": [
    "#### Comments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70642ef4-0179-419c-b7b2-575fb2578555",
   "metadata": {},
   "source": [
    "You might have noticed that there are some lines in our program that start with a hash mark (`#`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f39c0383-f2a8-4b06-bde2-e333a08abba6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Defining variables we're going to need\n",
    "#Giving the path to the text we're going to analyze\n",
    "filepath_of_text = \"gilman.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d53bd098-ea14-4b12-9293-2ef8bb2e085d",
   "metadata": {},
   "source": [
    "These are _comments_. Comments allow us to insert comments directly in the code without disrupting the running of the code (lines starting with the hash mark will be ignored by the code). Comments  can be used to explain what different aspects of the code are doing or as a way to write in or blank out different aspects of the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac002e56-1e31-4922-9e7f-d187ae9b312c",
   "metadata": {},
   "outputs": [],
   "source": [
    "marx = \"All that is solid melts into air, all that is holy is profaned, and man is at last compelled to face with sober senses his real conditions of life and his relations with his kind.\"\n",
    "\n",
    "#tokenized_marx = re.split(\"\\W+\", marx)\n",
    "tokenized_marx = marx.split()\n",
    "print(tokenized_marx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7dc0005-b293-43f5-9bd5-6574ec872378",
   "metadata": {},
   "outputs": [],
   "source": [
    "marx_without_punctuation = []\n",
    "for word in tokenized_marx:\n",
    "    if word.isalpha(): #the .isalpha() method checks whether a string is composed of alphabetic characters only\n",
    "        marx_without_punctuation.append(word)\n",
    "print(marx_without_punctuation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "401a3029-f6d7-4baa-9fa9-e6886ef021d2",
   "metadata": {},
   "source": [
    "You can see the flow of execution at work: code and comment can both be on a single line but the code has to come first to be executed. You could also toggle between the different splitting methods by commenting and uncommenting the different lines of code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d46adb5-90cc-4f44-805b-00f74087368b",
   "metadata": {},
   "source": [
    "Longer comments can be written using three quotation marks: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa2896c1-9f83-40e7-bf3c-4d535d810013",
   "metadata": {},
   "outputs": [],
   "source": [
    "marx = \"All that is solid melts into air, all that is holy is profaned, and man is at last compelled to face with sober senses his real conditions of life and his relations with his kind.\"\n",
    "\n",
    "#tokenized_marx = re.split(\"\\W+\", marx)\n",
    "tokenized_marx = marx.split()\n",
    "print(tokenized_marx)\n",
    "\n",
    "marx_without_punctuation = []\n",
    "for word in tokenized_marx:\n",
    "    if word.isalpha(): #the .isalpha() method checks whether a string is composed of alphabetic characters only\n",
    "        marx_without_punctuation.append(word)\n",
    "\n",
    "\"\"\" Another way of writing this for loop is by writing a list comprehension. This is just a more compact and precise way of writing\n",
    "marx_without_punctuation = [word for word in tokenized_marx if word.isalpha()] \"\"\"\n",
    "\n",
    "print(marx_without_punctuation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c4a6f69-3a66-4de4-a683-6216aec24345",
   "metadata": {},
   "source": [
    "<a id='section-15'></a>\n",
    "#### Working with Text Files in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11e496ec-ce9e-4101-8a59-eb501ba53700",
   "metadata": {},
   "source": [
    "<a id='section-17'></a>\n",
    "**File paths**\n",
    "\n",
    "In order to work with files, you need to be able to tell your program where the files are located. The location to the file is called the _path_. Inside the parantheses of `open()`, you insert the filepath of the file to be opened in quotation marks, as well as specifying which mode and encoding you want to open the file in. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3844da9d-32d1-4b84-aafe-af4db6309b2d",
   "metadata": {},
   "source": [
    "If the file is in the same folder as the notebook then the path to the file is simply the file name and extension (e.g. `gilman.txt`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd43f01f-e0cd-4f0d-a44d-ac98f2b223e4",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"gilman.txt\", mode=\"r\", encoding=\"utf-8\") as f:\n",
    "    raw_text = f.read()\n",
    "raw_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87ac11b2-6d5f-4ea6-9006-1d4f20d8caca",
   "metadata": {},
   "source": [
    "However, if your file is located somewhere else you will need to pass the path of the file to the `open()` function. \n",
    "\n",
    "If you double click the “gilman-corpus” directory, you can see under the search box that there is the icon of your home directory, then a slash, and then \"gilman-corpus\" (the name of the new directory where you are currently located). This is the path to the contents of the gilman directory. If you click back to the home directory, you move back one layer in the file structure to your home directory. Now let’s try specifying the path to this copy of the Yellow Wallpaper in the \"gilman-corpus\" directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d63d95a4-86d3-4b9f-beb4-12e318ecc82b",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "filepath_of_text = \"gilman-corpus/gilman_the-yellow-wallpaper.txt\"\n",
    "with open(filepath_of_text, mode=\"r\", encoding=\"utf-8\") as f:\n",
    "    full_text = f.read()\n",
    "full_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eb0bd7f-acf4-4520-be7a-91c949f4b3ea",
   "metadata": {},
   "source": [
    "Computers follow hierarchical systems of organization. Files contain other subfiles, which contain other subfiles etc. The filepath specifies the pathway through these files to a particular location. \n",
    "\n",
    "Paths can be _relative_ or _absolute_. Absolute filepaths provide the entire path to the current location from the home directory or root directory. Relative filepaths provide the location to the file based on where you are already located, i.e. based on current location or directory from which you are specifying the path. \n",
    "\n",
    "Tip: You can find the path to a file by right clicking on it and selecting “copy path”. These may not always be right for your code (it might give you the absolute path when you only need to relative path)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "639d8fa4-b53f-4354-9d20-421660f5bf7b",
   "metadata": {},
   "source": [
    "<a id='section-16'></a>\n",
    "**Opening and reading files**  \n",
    "\n",
    "In order to analyze text with Python you need to `open` the text in your program. This is done by using the function `open()` and a number of different _modes_ depending on what you want to do with the file (read it to analyze it, write to it, append to it, etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84c909fc-cddc-40ba-b635-5fab208dce00",
   "metadata": {},
   "source": [
    "- Reading a file with `.read()`\n",
    "\n",
    "When you `open()` a file using the `open()` function this creates a _file object_. Python is an _object-oriented_ programming language: it is built around entities called _objects_ that contain both data and methods for manipulating the data in the object. Once you create an object you can then manipulate it and interact with it with other objects. However, `open()` simply creates an object from your file, you need to specify in which mode you’re opening it in order to manipulate it accordingly. In our case, we need to `read()` the file object as text.\n",
    "\n",
    "Here are two ways of opening and reading a file. Note how we assign a variable to our open and read file so that we can conveniently refer to the open and read file by the variable name later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7bb220f-d7d4-4be9-8546-01a3b0922843",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "raw_text = open(\"gilman.txt\", mode=\"r\", encoding=\"utf-8\").read()\n",
    "raw_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "125ff648-0ae0-46b0-80b1-008c015e91c7",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"gilman.txt\") as f:\n",
    "    raw_text = f.read()\n",
    "raw_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "963fcfe8-4abc-4518-a909-1906222ca264",
   "metadata": {},
   "source": [
    "Opening a file with `with` means you don’t have to worry about closing the file (it will automatically close when the block is exited). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a734a639-ffce-4bd5-bd3f-738a85b06f39",
   "metadata": {},
   "source": [
    "- Writing to a file with `.write()`\n",
    "\n",
    "We could also open our file objects in write mode in order to write to a file. Note that writing to a file will overwrite the file if it already exists. If the file does not already exist, it creates a new file for writing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31e7a915-8a5d-41de-a85a-16d0042e0a9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"example_text.txt\", mode=\"w\") as f: \n",
    "    write_file = f.write(\"Hello file\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dca0df4-c882-46ca-866e-0da77b61552c",
   "metadata": {},
   "source": [
    "Can you see the newly created file in the list on the left? Double click to open it and see its contents."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43aeff27-98fd-48e5-aad1-75fd69462e37",
   "metadata": {},
   "source": [
    "For example, we could add a couple lines of code at the end of our program to write out the most frequent words to a text file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ab54b2d-4346-49d8-b700-b399faf2d79f",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"most-frequent-words-gilman.txt\", mode=\"w\") as file_object:\n",
    "    file_object.write(str(most_frequent_meaningful_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39e322c5-e7f5-45d5-b4fa-fc635d2828e1",
   "metadata": {},
   "source": [
    "- Append to a file with `.append()`\n",
    "\n",
    "If you want to write to a file, but you don’t want to overwrite its existing context you can use the `.append()` file method. This will add the text you want to add at the end of the existing file. If you append to a file that does not already exist, it simply create a new file for writing.\n",
    "\n",
    "Let’s add some text to our previous \"example_text.txt\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6cdaf0d-b6eb-473b-a09e-e33a421ee39e",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"example_text.txt\", mode=\"a\") as f:\n",
    "    append_file = f.write(\"\\n\" + \"how are you?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ee50c99-d72f-4c72-90d6-1deb9e3901bd",
   "metadata": {},
   "source": [
    "Double click to file \"example_text.txt\" to open it and see its contents. What happens if you run the append code multiple times?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f21cbf7-f1d4-4692-9bcb-342bdd43ae11",
   "metadata": {},
   "source": [
    "N.B. you can also simply write `\"a\"` or `\"w\"` or `\"r\"` instead of `mode=\"a\"`, `mode=\"w\"`, `mode=\"r\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7324bda4-b10f-423c-8a0f-f110497f850b",
   "metadata": {},
   "source": [
    "<a id='section-18'></a>\n",
    "**Working with multiple files**\n",
    "\n",
    "So far we have worked with only one text file, but we’ll want to work with many different text files in the future. You can work with multiple files with the help of the `Path` library or the `os` package."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaf3b9a2-8eb1-4ebb-a691-eade8b6cf19b",
   "metadata": {},
   "source": [
    "- Working with multiple files within a directory using the `Path` library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9fa1ff4-a65e-4eba-9941-4a72efac06d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "directory_path = \"gilman-corpus\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31d0015b-1c09-42f3-a5bb-38eecb3ef8a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Loop through all the files ending in .txt\n",
    "#Print the filepaths of each item\n",
    "for filepath in Path(directory_path).glob(\"*.txt\"): \n",
    "    print(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6d9d732-9df7-4f68-bb29-8382e24b53c6",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Loop through all files ending in .txt and open and read them\n",
    "#Print a slice of the first 20 characters \n",
    "#of each file in the gilman-corpus directory\n",
    "for filepath in Path(directory_path).glob(\"*.txt\"): \n",
    "    corpus = open(filepath, encoding=\"utf-8\").read()\n",
    "    print(corpus[:20]) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2df3f0b1-572b-45d4-a4e0-9331e596823f",
   "metadata": {},
   "source": [
    "- Working with multiple files using the `os` library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f43e4c25-a82a-4b3c-a9ff-77309b103436",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "# Loop through and print the name of files in a given directory\n",
    "gilman_corpus = os.listdir(\"gilman-corpus\")\n",
    "for file in gilman_corpus:\n",
    "    print(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de9bef1c-74a5-47b6-a110-b2ee496ffcff",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "# Loop through and print the name of files ending in.txt. (i.e. only text files) in a given directory\n",
    "gilman_corpus = os.listdir(\"gilman-corpus\")\n",
    "for file in gilman_corpus:\n",
    "    if file.endswith(\".txt\"):\n",
    "        corpus = open(\"gilman-corpus/\"+file, encoding=\"utf-8\").read()\n",
    "        print(corpus[:20])#This prints a slice of the first 20 characters of each file in the gilman_corpus directory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c75868ff-3c2e-420e-9904-cbf57f14e327",
   "metadata": {},
   "source": [
    "<a id='section-19'></a>\n",
    "#### Wrap up\n",
    "\n",
    "The aim of this course is that you become comfortable reading code and adapting it to your needs. It somewhat follows the ideas of “literate programming” or “literate computing” put forward by Donald Knuth in the 1970s that inspired to creation of Jupyter Notebooks. The idea of literate programming is that code can be written and presented in ways that highlight it as a human written cultural artifact — instead of prioritizing code as it is read by computers, literature programming wants to find ways of writing and reading code that make code more readable and understandable to human beings and that highlight the writer’s own thought processes. These ideas motivated the creation of Jupyter Notebooks to provide environments that can combine human prose with executable code. (cf. Knuth, Donald. _Literate Programming_. (1992) and this Programming Historian's [Introduction to Jupyter Notebooks](https://programminghistorian.org/en/lessons/jupyter-notebooks).)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2783a304-3454-4450-a1ec-01d9a43ee854",
   "metadata": {},
   "source": [
    "Read through our word counting program. Are you able to read through it and more or less follow what is going on now? How does it compare to when you first saw the program, are you able to understand more of it now?  \n",
    "\n",
    "Run the program and inspect the results. Do you find these most frequent words insightful or meaningful? Are some words strange or unexpected? What might account for some of these most frequent words? Open and skim through the source text (the `gilman.txt file`). Do some frequent words make more sense now?\n",
    "\n",
    "Try modifying and playing around with the program. Are there any words you might want filter out further from this list of most frequent words? How would you go about that?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1792a2ef-4689-484d-a590-47f9d7c6eb11",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Importing the libraries and modules we'll need\n",
    "import re\n",
    "from collections import Counter\n",
    "\n",
    "#Defining a function to split our text into words\n",
    "def split_into_words(any_chunk_of_text):\n",
    "    lowercase_text = any_chunk_of_text.lower()\n",
    "    split_words = re.split(\"\\W+\", lowercase_text)\n",
    "    return split_words\n",
    "\n",
    "#Defining variables we're going to need\n",
    "#Giving the path to the text we're going to analyze\n",
    "filepath_of_text = \"gilman.txt\"\n",
    "\n",
    "number_of_desired_words = 40\n",
    "\n",
    "#Defining a list of stopwords\n",
    "stopwords = ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours',\n",
    " 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers',\n",
    " 'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves',\n",
    " 'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are',\n",
    " 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does',\n",
    " 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until',\n",
    " 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into',\n",
    " 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down',\n",
    " 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here',\n",
    " 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',\n",
    " 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so',\n",
    " 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', 'should', 'now', 've', 'll', 'amp']\n",
    "\n",
    "#Opening and Reading the text to analyze it\n",
    "full_text = open(filepath_of_text, encoding=\"utf-8\").read()\n",
    "\n",
    "#Splitting the text into words by using our previously defined function\n",
    "all_the_words = split_into_words(full_text)\n",
    "\n",
    "#Making a list of words with our stopwords removed\n",
    "meaningful_words = []\n",
    "for word in all_the_words:\n",
    "        if word not in stopwords:\n",
    "            meaningful_words.append(word)\n",
    "\n",
    "\"\"\" Another way of writing this for loop\n",
    "meaningful_words = [word for word in all_the_words if word not in stopwords] \"\"\"\n",
    "\n",
    "#Counting how many times each word in our \"meaningful_words\" list appears\n",
    "meaningful_words_tally = Counter(meaningful_words)\n",
    "\n",
    "#Pulling out the top 40 most frequently occurring words from our complete tally using hte \"most_common\" method\n",
    "most_frequent_meaningful_words = meaningful_words_tally.most_common(number_of_desired_words)\n",
    "\n",
    "#Display these top 40 most frequent words\n",
    "most_frequent_meaningful_words\n",
    "\n",
    "#Writing the most frequent words to a file called \"most-frequent-words-gilman.txt\"\n",
    "#with open(\"most-frequent-words-gilman.txt\", mode=\"w\") as file_object:\n",
    "    #file_object.write(str(most_frequent_meaningful_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53f561f4-7ca6-4883-9d1b-ce8990bc9555",
   "metadata": {},
   "source": [
    "<a id='section-20'></a>\n",
    "### Debugging"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c235e153-b2d0-4410-8e8c-9da80f38a010",
   "metadata": {},
   "source": [
    "Just as you will become more and more familiar with reading code, you can also become more and more familiar with reading the errors that Python flag up to us when code doesn’t work. Python errors contain information to help us identify and fix the problem, they are a helpful resource that we can use to identify what is going wrong. \n",
    "\n",
    "It often take a series of attempts and some trial-and-error to get a program working. Errors can help us debug our programs along the way. \n",
    "\n",
    "There are different types of errors that give us an indication as to the type of problem we are dealing with:\n",
    "\n",
    "- _SyntaxError_: something is wrong with the Python syntax, the rules of writing Python code (i.e. the arrangements of words and punctuation in your code). It could be for example that you forgot to close a quotation mark, or that you forgot a colon after a `for` loop. If you are copy and pasting code, check that the quotation marks are the right kind of quotation mark that Python recognizes. In the error information, it will point out to you using a caret (^) where the error has occurred. \n",
    "\n",
    "- _NameError_: this error points to a problem with the name of a function or variable. For example, if a variable name cannot be found (maybe we forgot to run the cell that defines the variable, or maybe we misspelled the variable).\n",
    "\n",
    "- _TypeError_: this flags up issues of data type, when we are trying to perform an operation on the wrong kind of data type. Python might tell which data type is required in the error statement.\n",
    "\n",
    "- _AttributeError_: this error means that we trying to access something from an object that that object doesn’t possess or do something with an object that that object cannot do. For example, trying to use a method on a file that is a method for a string. \n",
    "\n",
    "- _FileNotFound Error_: this error means that the file name you have given cannot be located. This may be a problem of spelling the name wrong, or giving the wrong path to the file. \n",
    "\n",
    "\n",
    "When errors flag up, read through the code carefully, experiment with different things, talk it through with someone, and try googling and looking up forums (e.g. stackoverflow). There is a vibrant community around Python and you are probably not the first person to encounter the error you are encountering."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6eaf8d02-a88e-4257-aaba-205ccf06ce13",
   "metadata": {},
   "source": [
    "_Acknowledgements_: This notebook is heavily inspired by Melanie Walsh’s [_Introduction to Cultural Analytics & Python_](https://melaniewalsh.github.io/Intro-Cultural-Analytics/02-Python/00-Python.html) and also drew on the Constellate team's [tutorials](https://constellate.org/tutorials/), the Programming Historian's [tutorials](https://programminghistorian.org/en/lessons/working-with-text-files), and Ryan Heuser's [Literary text mining course](https://github.com/quadrismegistus/literarytextmining)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}