lab2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c406d3cd-1875-46ee-8291-6fe031ed3218",
   "metadata": {},
   "source": [
    "# Lab Assignment 2\n",
    "## Data Engineering 1\n",
    "### 20 points, Due Sunday, October 9 11:59pm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc4e4e04-77aa-4a55-8a38-04979d498c5d",
   "metadata": {},
   "source": [
    "(1) In class we began the GitHub workflow by creating a new GitHub repository from scratch on github.com. But another way to begin a new GitHub repository is to start by copying someone else's repository into your own account (this is called \"forking\" a repository). Follow these steps to initialize your repository for this lab:\n",
    "\n",
    "* Go to https://github.com/jkropko/lab2. Find the button marked Fork in the upper-right corner of the screen and push it. You will be taken to a new GitHub page that looks exactly the same, except that the URL now has your GitHub user name instead of \"jkropko\". (Side note: most open software has a public GitHub repo, and you can get your own copy of it by forking the repository. So for example, suppose you wish pandas had a more intuitive function for finding missing values in a data frame: you could fork https://github.com/pandas-dev/pandas and add your own code to your version of this repository to make your own custom version of pandas. If your code works extremely well, you can also suggest the original pandas repo incorporate your code by issuing a \"pull request\".)\n",
    "\n",
    "* On your computer, choose a location for working with the local files and navigate to that location in a terminal. Use the `git clone` command to download your repository and activate the git commands. (No need to create a \"lab2\" folder yourself -- one will be created when you use `git clone`.)\n",
    "\n",
    "* Use `cd` to enter the lab2 folder (but not the data folder inside this one). Then create three files:\n",
    "\n",
    "    * A requirements.txt file that installs jupyterlab==3.4.7, pandas==1.4.4, numpy==1.23.3, requests==2.28.1, and beautifulsoup4==4.8.1\n",
    "\n",
    "    * A Dockerfile that installs python:3.10.7-bullseye, copies the requirements.txt file into the container and runs pip install for the packages in the requirements.txt file, sets a default working directory inside the container with an appropriate name, exposes port 8888 and launches jupyter lab.\n",
    "    \n",
    "    * A .env file. It can be empty for now, but you will add API keys to this file.\n",
    "    \n",
    "* Use the `git add`, `git commit`, and `git push` commands to save these files to your GitHub repository. (Note that I already added a .gitignore file to the repository which will tell git not to upload your .env file)\n",
    "\n",
    "* Build the Docker image from your Dockerfile, then run the Docker container from this image. Be careful to specify a port for running Jupyter Lab locally, attach the local folder as a volume to the container's working directory, and specify the .env file. \n",
    "\n",
    "* Use the containerized version of Jupyter Lab to work on the rest of this lab. Save your notebook in the container, and add, commit, and push this notebook to GitHub as well.\n",
    "\n",
    "To receive credit for this problem, just type the URL of your GitHub repository in your notebook. [3 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87c26807-7142-487c-9028-bffb9ab8e207",
   "metadata": {},
   "source": [
    "(2) Now that you have a local copy of your GitHub repository, notice that there is a data folder. Inside this folder are 11 flat files, each containing another version of the same dataset. Only one of the files, `data_clean.csv`, can be loaded properly with a straightforward use of `pd.read_csv()`. The other 10 files all need either an argument within `pd.read_csv()` or a different pandas method to load properly. Find the code that correctly loads each of the 10 files. Display the head of each data frame to confirm that it is properly loaded. \n",
    "\n",
    "(Note: all of the arguments and methods you need are discussed in https://jkropko.github.io/surfing-the-data-pipeline/ch2.html. Don't worry about small differences in the column names, such as capitalization, if it remains clear which column is which. A couple of the data files have more missing data than `data_clean.csv`, which is fine so long as the missing values are coded as missing, and not as numeric values.) [3 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73605216-6eee-4bff-91a9-38a10d8ec243",
   "metadata": {},
   "source": [
    "(3) For this problem, you will be accessing the open APIs maintained by NASA. All of the API endpoints share a credentialing system with the same API key. First, register for an API key here: https://api.nasa.gov/. Once you have the key, save it in your .env file. \n",
    "\n",
    "Then click on Browse APIs and find the information for the Asteroids - NeoWs API. This API reports data on \"near Earth objects\" in space, such as asteroids. It reports the size of the objects, their speed, direction, and distance to Earth. It even codes whether or not NASA considers the object to be potentially hazardous to Earth.\n",
    "\n",
    "Write the code to access this API and generate a single data frame with all of the near Earth objects reported by NASA over the last 7 days.\n",
    "\n",
    "To receive full credit for this problem, make sure you\n",
    "\n",
    "* Supply an accurate user-agent in the headers along with your email address in the 'From' field\n",
    "\n",
    "* Provide your NASA API key in the way the documentation instructs\n",
    "\n",
    "* Use `pd.json_normalize()` and `pd.concat()` to extract and combine several data frames from the JSON output.\n",
    "\n",
    "Note: in your output, the `close_approach_data` field will still be in a dictionary within a list in the final data frame. That's okay for this problem. If you want to extract those data and store them as additional columns, please do: I recommend using list comprehensions as follows:\n",
    "```\n",
    "df['close_approach_date'] = [d[0][''close_approach_date''] for d in df['close_approach_data']]\n",
    "```\n",
    "[3 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee77ca9f-55d4-48e5-b319-d9490b70c10a",
   "metadata": {},
   "source": [
    "(4) Read the following blog post about how to read a robots.txt file: https://yoast.com/ultimate-guide-robots-txt/ Then look at the robots.txt files for the following websites, and report whether they allow the following kinds of web-scraping. For each question, explain your answer and copy-and-paste the relevant parts of the robots.txt file if necessary:\n",
    "\n",
    "a. Pulling the current listed prices for houses for sale in Charlottesville from zillow.com\n",
    "\n",
    "b. Scraping current stock prices off of google.com/finance\n",
    "\n",
    "c. Copying the list of twitter accounts that each NBA player follows\n",
    "\n",
    "d. The lyrics to Lizzo's Good as Hell from https://genius.com/Lizzo-good-as-hell-lyrics\n",
    "\n",
    "[2 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7683f75-e7fe-4a73-8689-7ac297a19e59",
   "metadata": {},
   "source": [
    "(5) In class we accessed the Genius API to find the most popular songs by Lizzo. But the Genius API does not allow us to download the lyrics to any songs. This looks like a job for web scraping. \n",
    "\n",
    "Good As Hell is the 15th track on Lizzo's 2019 album Cuz I Love You. In this problem, you will web-scrape genius.com to bring the lyrics to Good As Hell into Python, find links to the lyrics to the other songs on the Cuz I Love You album, and build a spider to extract all the song lyrics from this album.\n",
    "\n",
    "A few tricks I found useful to complete these tasks:\n",
    "\n",
    "** Trick 1**: A list comprehension is a for-loop across items of a list that places the outputs into another list. The general syntax is\n",
    "```\n",
    "[f(x) for x in list]\n",
    "```\n",
    "where `list` is the list of input values, `x` is a token that stands in for one general element of the input list, and `f(x)` is what we want to do to each element. For example, if this is my list of inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "59ae8d79-45fb-4844-83ee-f485e2c1e6d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = [3,5,8,9,12]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d164987-cddd-4f81-90a2-634a034bd302",
   "metadata": {},
   "source": [
    "and I want to square then subtract 1 from each element, I can use the syntax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "87ef370c-e63f-4911-beff-e740e1c75438",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[8, 24, 63, 80, 143]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(x**2) - 1 for x in inputs]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dce70e2c-a6e2-4da9-a4fe-12d9a097a1d3",
   "metadata": {},
   "source": [
    "For this problem, you will need to use a *filtered* list comprehension, which uses `if` to filter down the list of outputs. For example, if I only want to keep the even input numbers in this list (8 and 12), I can use `if x/2 == round(x/2)` (no decimals when dividing by 2) to narrow the list down:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d80d2459-a7f8-43b1-94ab-0d3444839d49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[63, 143]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(x**2) - 1 for x in inputs if x/2 == round(x/2)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dab915f-bf44-4a3f-8887-c3d5fadff6e2",
   "metadata": {},
   "source": [
    "**Trick 2**: The second trick that will be needed is the `attrs{}` argument of BeautifulSoup's `.find_all()` method. When searching for particular HTML tags, you may want to narrow down the search to only those tags with a particular attribute. For example, if I saved the parsed HTML as `mysoup`, and I want to find all the `a` tags that also have an attribute `rel=\"noopener\"`, I can type\n",
    "```\n",
    "mysoup.find_all('a', attrs{'rel':'noopener'})\n",
    "```\n",
    "If the attribute has no hyphens, and is not a Python reserved variable like `name` or `class`, then a shortcut for the same function is\n",
    "```\n",
    "mysoup.find_all('a', rel='noopener')\n",
    "```\n",
    "but if the attribute name hyphens or is a reserved variable, then the `attrs{}` approach still works.\n",
    "\n",
    "**Trick 3**: Finally, suppose you have one text document in a way in which each paragraph is a separate item of a list. The `.join()` method can combine all these elements into a single string. For example, if I have a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e0bbef60-4d93-4549-a3b3-4d6b66aaae30",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I',\n",
       " 'do',\n",
       " 'my',\n",
       " 'hair',\n",
       " 'toss,',\n",
       " 'check',\n",
       " 'my',\n",
       " 'nails',\n",
       " 'Baby,',\n",
       " 'how',\n",
       " 'you',\n",
       " 'feelin?',\n",
       " 'Feelin',\n",
       " 'good',\n",
       " 'as',\n",
       " 'hell']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = ['I','do','my','hair','toss,', 'check','my','nails','Baby,','how','you','feelin?','Feelin','good','as','hell']\n",
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f995215-4068-4127-a4c2-428a2e2f4d6a",
   "metadata": {},
   "source": [
    "and I want to combine them into a single string in which each word is separated by a space, I type a space in quotes, then apply the `.join(words)` method to it:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cd6d46e3-738d-4a02-ace6-09bbf068d79e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I do my hair toss, check my nails Baby, how you feelin? Feelin good as hell'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ed3f4e-0d80-43f5-ae2d-9d41e0c4da7b",
   "metadata": {},
   "source": [
    "Using these three tricks, perform the following tasks:\n",
    "\n",
    "(a) Use `requests` and `BeautifulSoup` to scrape the lyrics to Good As Hell from https://genius.com/Lizzo-good-as-hell-lyrics\n",
    "\n",
    "(b) Notice that at towards the bottom of https://genius.com/Lizzo-good-as-hell-lyrics there is a menu containing a track list for the Cuz I Love You album. Each link here takes you to the lyrics for that song. Use `requests` and `BeautifulSoup` to save the list of song lyric URLs in a Python list.\n",
    "\n",
    "(c) Build a spider: first write a function that applies the code you wrote for part (a) to a user-supplied URL; then apply this function to each of the URLs in the list you constructed in part (b).\n",
    "\n",
    "[5 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd37c304-78f7-46a3-993d-1d5e32c68e56",
   "metadata": {},
   "source": [
    "(6) Finding the hidden API:\n",
    "\n",
    "APIs are the primary mechanism for transfering data over the internet, but most APIs are for internal use for a website and are not intended for outside users. When this is the case, there won't be an obvious link to the API and there won't be any documentation. You can still sometimes get access to the API. This exercise will guide you through one instance of finding and using a hidden API.\n",
    "\n",
    "It's the time of year for Spirit Halloween stores to pop up. They often take over the stores where large chains have recently met their demise. Go to the store locator page: https://stores.spirithalloween.com/ Notice that the top hit says \"Former Sears\". I want to know how often Sears appears in the descriptions of these stores, what other dead chains show up, and how frequently. If you right click on this page and view source, the data that appears about the nearby stores does not appear. Instead this website is calling a hidden API that we can access.\n",
    "\n",
    "For this problem, use the webpage inspector in the Mozilla Firefox web browser. If you don't have Firefox, download it here: https://www.mozilla.org/en-US/firefox/new/\n",
    "\n",
    "**Step 1**: With Firefox, go to https://stores.spirithalloween.com/ Right click on this page, and select \"Inspect\". (You can still use Chrome or anything you want for Jupyter Lab, just have Firefox open for the inspect tool).\n",
    "\n",
    "**Step 2**: Inspect is a complicated but extremely useful tool. It reports all APIs and other web-based connections that a website makes. Click on Network to see these connections. These are all the calls the Spirit Halloween website makes to various APIs to display images, maps, ads, and addresses. \n",
    "\n",
    "**Step 3**: In the right-hand window within the Inspect tool, click on Response. Our task is to sift through the various API calls and to look at the responses until we see the specific address information we need. This took me a long, long time, and I'll save you that trouble -- find the Domain maps.spirithalloween.com, click it, and look at the JSON that appears under Response. Scroll down until you see the addresses and the phrase \"Former Sears\". If you do not see maps.spirithalloween.com under Domains, reload the page and look again.\n",
    "\n",
    "**Step 4**: Now that we know the API call that Spirit Halloween used to get the addresses, we can deduce the root, endpoint, and some of the parameters. Hover your mouse over file. It displays: https://maps.spirithalloween.com/api/getAsyncLocations?template=domain&level=domain. So the root should be https://maps.spirithalloween.com, the endpoint should be /api/getAsyncLocations, and two of the parameters should be template and level, both set equal to \"domain\".\n",
    "\n",
    "**Step 5**: Take a closer look at Response. There are other parameters here we can use. At the bottom of the JSON output is a key named \"options\". I didn't know this for sure, but my bet was that the key-value pairs inside \"options\" can be changed as parameters in the call to the API. My goal is to get all of the Spirit Halloween stores, not just the ones near me. So some of the values I wanted to change are lat and lng, which define my location, radius which defines the distance in miles from my location, and limit, which I bet specifies the maximum number of results. I want these values to be 'lat': 40.380028, 'lng': -97.910156, which places my location in the middle of the country, 'radius': 1800 which captures the entire lower 48 states, and 'limit': 1500 which exceeds the number of Spirit Halloween stores (I had to guess-and-check that).\n",
    "\n",
    "For this problem, take it from here. Issue an API call to https://maps.spirithalloween.com with endpoint /api/getAsyncLocations. Provide your user-agent and email address in the headers, and set the parameters as discussed in step 5. Find a way to extract the descriptive phrases such as 'Former Sears' and 'Next to Ashley Furniture' from each store's address and store them in a list. Filter this list to only the elements that contain the word \"Former\". Then report the frequencies. If you saved these phrases as a list named `formerlocs` you can get frequencies by typing:\n",
    "```\n",
    "formerlocsDF = pd.DataFrame()\n",
    "formerlocsDF['formerlocs'] = formerlocs\n",
    "formerlocsDF['formerlocs'].value_counts()\n",
    "```\n",
    "Hint: even though the output is JSON, some of the data inside the JSON is encoded as HTML, so you will need to use `BeautifulSoup` as well as the `json` package. [4 points]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}