Web_Scraping,_Regular_Expressions,_and_Visualization_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Web Scraping, Regular Expressions, and Visualization .ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/soiqualang/colab/blob/master/Web_Scraping%2C_Regular_Expressions%2C_and_Visualization_.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8F-vlK3uAWH1",
        "colab_type": "text"
      },
      "source": [
        "# Introduction\n",
        "\n",
        "This notebook is meant to serve as a small example project of how to get data from the web using Python. Here we will scrape the data from the web, parse the results using regular expressions, and visualize the data. This small project could probably be done a lot more efficiently by hand, but the ideas here are powerful and can be extended to much larger applications. There are many tools that a data scientist will need to use, and web scraping, regular expressions, and visualization are all good techniques to practice! "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YuG1JtCLBU-I",
        "colab_type": "text"
      },
      "source": [
        "# Web Scraping Using Requests and Beautiful Soup (bs4)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nAl0C2yPFita",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# requests for fetching html of website\n",
        "import requests\n",
        "\n",
        "# Make the request to a url\n",
        "r = requests.get('http://www.cleveland.com/metro/index.ssf/2017/12/case_western_reserve_university_president_barbara_snyders_base_salary_and_bonus_pay_tops_among_private_colleges_in_ohio.html')\n",
        "\n",
        "# Create soup from content of request\n",
        "c = r.content\n",
        "\n",
        "from bs4 import BeautifulSoup\n",
        "\n",
        "soup = BeautifulSoup(c)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YaPgkW1ZBq8e",
        "colab_type": "text"
      },
      "source": [
        "We can find an element on the page by inspecting the page (right click and hit inspect element). We then use a series of HTML selectors to find the appropriate tags which contain the content we are interested in. The next code block finds the main text of the entire article. We will then further subset the text to the relevant table and save it as a text object. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "owzwg4U1JWwa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Find the element on the webpage\n",
        "main_content = soup.find('div', attrs = {'class': 'entry-content'})\n",
        "main_content"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bTYLGSParbOH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Extract the relevant information\n",
        "content = main_content.find('ul').text\n",
        "\n",
        "import pprint\n",
        "pprint.pprint(content)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vusj0xMpCNdn",
        "colab_type": "text"
      },
      "source": [
        "We know have the data we need to focus on as a text string. The next step is to parse this information using regular expressions to identify the presidents, colleges, and salaries. Regular Expressions are intimidating at first, and require practice to learn to use effectively. I am definitely not an expert, and am using this project partly to get more familiar with regular expressions. The best place to get started is simply with the [Python Documentation](https://docs.python.org/3/library/re.html) for the `re` library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VMeuScw5szZl",
        "colab_type": "text"
      },
      "source": [
        "# Regular Expressions Using `re`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9QE7mDVGs2vo",
        "colab_type": "text"
      },
      "source": [
        "## Presidents\n",
        "\n",
        "First we want to extract the names of the presidents. We do this taking advantage of the fact that the names come at the beginning of each newline. We therefore use an expression in the multiline mode, which treats every newline character as the end of the string (so the next character is the start of a string). The names also all end with a comma, so we can use this to bound the names. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U0MUZA5eOy9s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re\n",
        "\n",
        "# Create a pattern to match names\n",
        "name_pattern = re.compile(r'^([A-Z]{1}.+?)(?:,)', flags = re.M)\n",
        "name_pattern.findall(content)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7wL_BHOGQC2w",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "names = name_pattern.findall(content)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SnfSRd9ks9Aa",
        "colab_type": "text"
      },
      "source": [
        "## Colleges\n",
        "\n",
        "The next piece of information to extract is the name of the schools. We can use the fact that each school name is preceded either by a comma (Kenyon College) or a comma and a space (I don't think this was intentional, but we can handle it). Each school name ends with a colon `:` or with a space and a left parenthesis ` (` or with a comma `,`. We use both of these to bound the college expression and extract the useful information. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BpI3uEHoP3X-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Remind ourselves what our soup looks like\n",
        "pprint.pprint(content)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LiDpUp7IJ8LY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Make school patttern and examine results\n",
        "school_pattern = re.compile(r'(?:,|,\\s)([A-Z]{1}.*?)(?:\\s\\(|:|,)')\n",
        "school_pattern.findall(content)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qVArEMqRRhcs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Extract the schools\n",
        "schools = school_pattern.findall(content)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UH8jq4O8tI7c",
        "colab_type": "text"
      },
      "source": [
        "## Salaries\n",
        "\n",
        "Finally, we need to get the salaries. This is relatively easy because all of the salaries are preceded by a dollar sign. Once we have extract the salaries as strings, we can use a Python list comprehension to remove the $ , and convert the string to a float. This uses a few Python shortcuts, and I like how elegant the list comprehension is. Writing Python can really a joy! "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tcIb831uSiii",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Pattern to match the salaries\n",
        "salary_pattern = re.compile(r'\\$.+')\n",
        "salary_pattern.findall(content)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9cMR7Z_2tSSw",
        "colab_type": "text"
      },
      "source": [
        "### Converting Dollar Strings to Numbers\n",
        "\n",
        "First we can see a brief example of each of the steps in the list comprehension. We use the `split` method to split each string into two separate strings at the comma (starting with the first character after the $). Then, we `join`, the two strings together with no separating character and convert the result to a float. All of this is wrapped in a list comprehension. The end result is a list of numeric values representing the Presidents' salaries."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P4f2TmO2E-D4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Messy salary\n",
        "salary = '$876,001'\n",
        "\n",
        "# Exclude the $ and split the string on the comma\n",
        "salary[1:].split(',')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Hndq4TAFJWI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Same operation but now join the list with no space\n",
        "''.join(salary[1:].split(','))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XKzwukqRFXvY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Finally convert the string to a float\n",
        "float(''.join(salary[1:].split(',')))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Kf6u5poFdNQ",
        "colab_type": "text"
      },
      "source": [
        "#### Example List Comprehension to Test Method"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S9rfMCTyVWbQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Messy salaries\n",
        "salaries = ['$876,001', '$543,903', '$2453,896']\n",
        "\n",
        "# Convert salaries to numbers using the above procedure in a list comprehension \n",
        "[int(''.join(s[1:].split(','))) for s in salaries]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O6-YQ_wjT8Ji",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Extract all the salaries and convert to integers\n",
        "salaries = salary_pattern.findall(content)\n",
        "\n",
        "# List comprehension to convert strings to floats\n",
        "salaries = [int(''.join(s[1:].split(','))) for s in salaries]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f84SCQCzUaim",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "salaries"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Z40AFYOP1_M",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Sanity check to make sure everything is correct!\n",
        "len(names) == len(schools) == len(salaries)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SkpO7ODMte0D",
        "colab_type": "text"
      },
      "source": [
        "# Visualization \n",
        "\n",
        "We will use the `matplotlib` and `seaborn` libraries for visualizing the results. `matplotlib` is great for creating quick visualizations, and I like the aesthetic style of `seaborn`. We'll start off by storing the data in a `pandas` dataframe, the common data structure of choice for data science.\n",
        "\n",
        "Here I am manually adding in my President's information to the dataframe. Sometimes knowing when it is faster to just do something by hand rather than writing a complicated program is a vital skill in data science."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y25EZdiHUmt6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Put information into a dataframe\n",
        "df = pd.DataFrame({'salary': salaries, \n",
        "                   'President': names,\n",
        "                   'College': schools})\n",
        "\n",
        "# Append information\n",
        "df.loc[17, :] = ['CWRU', 'Barbara Synder', 1154000]\n",
        "\n",
        "# Sort the values by highest to lowest salary\n",
        "df = df.sort_values('salary', ascending=False).reset_index().drop(columns='index')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j887NjcRt0No",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6thdceSxt9-0",
        "colab_type": "text"
      },
      "source": [
        "## Quick First Visualization\n",
        "\n",
        "We can use plotting functionality built into pandas to rapidly create an initial figure. This at least conveys the information although it does not look very nice! "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4zc1dqY0ZlrS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.plot(kind='barh', x = 'President', y = 'salary');"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HYD2lStbuChr",
        "colab_type": "text"
      },
      "source": [
        "## Improve the Plot\n",
        "\n",
        "Now comes an iterative procedure of improving the graphic. A lot of this involves using Stack Overflow and the [seaborn documentation](https://seaborn.pydata.org/generated/seaborn.barplot.html) to figure out how to make the plot look exactly like we want it. Plotting syntax often is pretty complicated, but don't worry about the specifics. You will always be able to look these up or build on old plots you or others made."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LvceG6InaGDU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "# Pick a style\n",
        "plt.style.use('fivethirtyeight')\n",
        "plt.rcParams['font.size'] = 16\n",
        "\n",
        "import seaborn as sns\n",
        "\n",
        "# Sort the values by highest to lowest salary\n",
        "df = df.sort_values('salary', ascending=False).reset_index()\n",
        "\n",
        "# Shorten this one name for plotting\n",
        "df.ix[df['College'] == 'University of Mount Union', 'College'] = 'Mount Union'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KHeNb8oLaKSe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create the basic figure\n",
        "plt.figure(figsize=(10, 8))\n",
        "sns.barplot(x = 'salary', y = 'President', data = df, \n",
        "            color = 'tomato', edgecolor = 'k', linewidth = 2)\n",
        "\n",
        "# Add text showing values and colleges\n",
        "for i, row in df.iterrows():\n",
        "  plt.text(x = row['salary'] + 6000, y = i + 0.15, s = '$%d' % (round(row['salary'] / 1000) * 1000))\n",
        "  plt.text(x = 5000, y = i + 0.15, s = row['College'], size = 14)\n",
        "\n",
        "# Labels are a must!\n",
        "plt.xticks(size = 16); plt.yticks(size = 18)\n",
        "plt.xlabel('Total Compensation ($)')\n",
        "plt.ylabel('President') \n",
        "plt.title('2015 Compensation of Private Ohio College Presidents');"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NEO_bUhSNuWT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Calculate value of 5 minutes of your presidents time\n",
        "five_minutes_fraction = 5 / (2000 * 60)\n",
        "total_df = pd.DataFrame(df.groupby('College')['salary'].sum())\n",
        "total_df['five_minutes_cost'] = round(total_df['salary'] * five_minutes_fraction)\n",
        "total_df = total_df.sort_values('five_minutes_cost', ascending = False).reset_index()\n",
        "\n",
        "total_df"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ctCFhJPlZxAs",
        "colab_type": "text"
      },
      "source": [
        "## Final Product\n",
        "\n",
        "After several attempts (I have not shown all my failures along the way), we can create the final plot: how much are you paying for five minutes of your president's time. I wouldn't say the plot is production quality, but it is a good ending point for this project!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wIYvBEASRGnX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Text for caption\n",
        "txt = 'Calculated from 2015 Total Compensation assuming 2000 hrs worked/year. Source: Chronical of Higher Education'\n",
        "\n",
        "# Create the basic barplot\n",
        "plt.figure(figsize=(10, 8))\n",
        "sns.barplot(x = 'five_minutes_cost', y = 'College', data = total_df, \n",
        "            color = 'red', edgecolor = 'k', linewidth = 2)\n",
        "\n",
        "# Add the text with the value\n",
        "for i, row in total_df.iterrows():\n",
        "  plt.text(x = row['five_minutes_cost'] + 0.5, y = i + 0.15, \n",
        "           s = '$%d' % (row['five_minutes_cost']), size = 18)\n",
        "\n",
        "# Add the caption\n",
        "plt.text(x = -5, y = 20, s = txt, size = 14)\n",
        "\n",
        "# Add the labels\n",
        "plt.xticks(size = 16); plt.yticks(size = 18)\n",
        "plt.xlabel('Value ($)')\n",
        "plt.ylabel('') \n",
        "plt.title(\"Value of Five Minutes of Your President's Time\");"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JDt9iuHnV8qy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}