lab2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c406d3cd-1875-46ee-8291-6fe031ed3218",
   "metadata": {},
   "source": [
    "# Lab Assignment 2\n",
    "## Data Engineering 1\n",
    "### 20 points, Due Sunday, October 9 11:59pm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc4e4e04-77aa-4a55-8a38-04979d498c5d",
   "metadata": {},
   "source": [
    "(1) In class we began the GitHub workflow by creating a new GitHub repository from scratch on github.com. But another way to begin a new GitHub repository is to start by copying someone else's repository into your own account (this is called \"forking\" a repository). Follow these steps to initialize your repository for this lab:\n",
    "\n",
    "* Go to https://github.com/jkropko/lab2. Find the button marked Fork in the upper-right corner of the screen and push it. You will be taken to a new GitHub page that looks exactly the same, except that the URL now has your GitHub user name instead of \"jkropko\". (Side note: most open software has a public GitHub repo, and you can get your own copy of it by forking the repository. So for example, suppose you wish pandas had a more intuitive function for finding missing values in a data frame: you could fork https://github.com/pandas-dev/pandas and add your own code to your version of this repository to make your own custom version of pandas. If your code works extremely well, you can also suggest the original pandas repo incorporate your code by issuing a \"pull request\".)\n",
    "\n",
    "* On your computer, choose a location for working with the local files and navigate to that location in a terminal. Use the `git clone` command to download your repository and activate the git commands. (No need to create a \"lab2\" folder yourself -- one will be created when you use `git clone`.)\n",
    "\n",
    "* Use `cd` to enter the lab2 folder (but not the data folder inside this one). Then create three files:\n",
    "\n",
    "    * A requirements.txt file that installs jupyterlab==3.4.7, pandas==1.4.4, numpy==1.23.3, requests==2.28.1, and beautifulsoup4==4.8.1\n",
    "\n",
    "    * A Dockerfile that installs python:3.10.7-bullseye, copies the requirements.txt file into the container and runs pip install for the packages in the requirements.txt file, sets a default working directory inside the container with an appropriate name, exposes port 8888 and launches jupyter lab.\n",
    "    \n",
    "    * A .env file. It can be empty for now, but you will add API keys to this file.\n",
    "    \n",
    "* Use the `git add`, `git commit`, and `git push` commands to save these files to your GitHub repository. (Note that I already added a .gitignore file to the repository which will tell git not to upload your .env file)\n",
    "\n",
    "* Build the Docker image from your Dockerfile, then run the Docker container from this image. Be careful to specify a port for running Jupyter Lab locally, attach the local folder as a volume to the container's working directory, and specify the .env file. \n",
    "\n",
    "* Use the containerized version of Jupyter Lab to work on the rest of this lab. Save your notebook in the container, and add, commit, and push this notebook to GitHub as well.\n",
    "\n",
    "To receive credit for this problem, just type the URL of your GitHub repository in your notebook. [3 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c546fe7-bcad-49ba-bc70-1467e06c30b0",
   "metadata": {},
   "source": [
    "https://github.com/beauleblond/lab2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87c26807-7142-487c-9028-bffb9ab8e207",
   "metadata": {},
   "source": [
    "(2) Now that you have a local copy of your GitHub repository, notice that there is a data folder. Inside this folder are 11 flat files, each containing another version of the same dataset. Only one of the files, `data_clean.csv`, can be loaded properly with a straightforward use of `pd.read_csv()`. The other 10 files all need either an argument within `pd.read_csv()` or a different pandas method to load properly. Find the code that correctly loads each of the 10 files. Display the head of each data frame to confirm that it is properly loaded. \n",
    "\n",
    "(Note: all of the arguments and methods you need are discussed in https://jkropko.github.io/surfing-the-data-pipeline/ch2.html. Don't worry about small differences in the column names, such as capitalization, if it remains clear which column is which. A couple of the data files have more missing data than `data_clean.csv`, which is fine so long as the missing values are coded as missing, and not as numeric values.) [3 points]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ccf8da74-2be8-43e6-bf84-5239687d5a65",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import json\n",
    "import os\n",
    "import collections\n",
    "collections.Callable = collections.abc.Callable\n",
    "oldpath = os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d7199db8-9963-4078-9631-22ceadecf4f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(\"data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e22d82a7-a663-4a22-b882-5b05f3218c3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>Happiness score</th>\n",
       "      <th>Whisker-high</th>\n",
       "      <th>Whisker-low</th>\n",
       "      <th>Dystopia (1.92) + residual</th>\n",
       "      <th>Explained by: GDP per capita</th>\n",
       "      <th>Explained by: Social support</th>\n",
       "      <th>Explained by: Healthy life expectancy</th>\n",
       "      <th>Explained by: Freedom to make life choices</th>\n",
       "      <th>Explained by: Generosity</th>\n",
       "      <th>Explained by: Perceptions of corruption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Finland</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>1.305</td>\n",
       "      <td>1.592</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Norway</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>2.383</td>\n",
       "      <td>1.456</td>\n",
       "      <td>1.582</td>\n",
       "      <td>0.861</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>7.398</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Country  Happiness score  Whisker-high  Whisker-low  \\\n",
       "0      Finland            7.632         7.695        7.569   \n",
       "1       Norway            7.594         7.657        7.530   \n",
       "2      Denmark            7.555         7.623        7.487   \n",
       "3      Iceland            7.495         7.593        7.398   \n",
       "4  Switzerland            7.487         7.570        7.405   \n",
       "\n",
       "   Dystopia (1.92) + residual  Explained by: GDP per capita  \\\n",
       "0                       2.595                         1.305   \n",
       "1                       2.383                         1.456   \n",
       "2                       2.370                         1.351   \n",
       "3                       2.426                         1.343   \n",
       "4                       2.320                         1.420   \n",
       "\n",
       "   Explained by: Social support  Explained by: Healthy life expectancy  \\\n",
       "0                         1.592                                  0.874   \n",
       "1                         1.582                                  0.861   \n",
       "2                         1.590                                  0.868   \n",
       "3                         1.644                                  0.914   \n",
       "4                         1.549                                  0.927   \n",
       "\n",
       "   Explained by: Freedom to make life choices  Explained by: Generosity  \\\n",
       "0                                       0.681                     0.192   \n",
       "1                                       0.686                     0.286   \n",
       "2                                       0.683                     0.284   \n",
       "3                                       0.677                     0.353   \n",
       "4                                       0.660                     0.256   \n",
       "\n",
       "   Explained by: Perceptions of corruption  \n",
       "0                                    0.393  \n",
       "1                                    0.340  \n",
       "2                                    0.408  \n",
       "3                                    0.138  \n",
       "4                                    0.357  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_clean = pd.read_csv('data_clean.csv')\n",
    "data_clean.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ee87408a-5bcc-4e20-a6f5-9508e5f965df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>Happiness score</th>\n",
       "      <th>Whisker-high</th>\n",
       "      <th>Whisker-low</th>\n",
       "      <th>Dystopia (1.92) + residual</th>\n",
       "      <th>Explained by: GDP per capita</th>\n",
       "      <th>Explained by: Social support</th>\n",
       "      <th>Explained by: Healthy life expectancy</th>\n",
       "      <th>Explained by: Freedom to make life choices</th>\n",
       "      <th>Explained by: Generosity</th>\n",
       "      <th>Explained by: Perceptions of corruption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Finland</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>1.305</td>\n",
       "      <td>1.592</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Norway</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>2.383</td>\n",
       "      <td>1.456</td>\n",
       "      <td>1.582</td>\n",
       "      <td>0.861</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>7.398</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Country  Happiness score  Whisker-high  Whisker-low  \\\n",
       "0      Finland            7.632         7.695        7.569   \n",
       "1       Norway            7.594         7.657        7.530   \n",
       "2      Denmark            7.555         7.623        7.487   \n",
       "3      Iceland            7.495         7.593        7.398   \n",
       "4  Switzerland            7.487         7.570        7.405   \n",
       "\n",
       "   Dystopia (1.92) + residual  Explained by: GDP per capita  \\\n",
       "0                       2.595                         1.305   \n",
       "1                       2.383                         1.456   \n",
       "2                       2.370                         1.351   \n",
       "3                       2.426                         1.343   \n",
       "4                       2.320                         1.420   \n",
       "\n",
       "   Explained by: Social support  Explained by: Healthy life expectancy  \\\n",
       "0                         1.592                                  0.874   \n",
       "1                         1.582                                  0.861   \n",
       "2                         1.590                                  0.868   \n",
       "3                         1.644                                  0.914   \n",
       "4                         1.549                                  0.927   \n",
       "\n",
       "   Explained by: Freedom to make life choices  Explained by: Generosity  \\\n",
       "0                                       0.681                     0.192   \n",
       "1                                       0.686                     0.286   \n",
       "2                                       0.683                     0.284   \n",
       "3                                       0.677                     0.353   \n",
       "4                                       0.660                     0.256   \n",
       "\n",
       "   Explained by: Perceptions of corruption  \n",
       "0                                    0.393  \n",
       "1                                    0.340  \n",
       "2                                    0.408  \n",
       "3                                    0.138  \n",
       "4                                    0.357  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data1 = pd.read_csv('data1.csv', header=2)\n",
    "data1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9b99e568-7f03-4bee-a966-cd5262d8cd36",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>Happiness score</th>\n",
       "      <th>Whisker-high</th>\n",
       "      <th>Whisker-low</th>\n",
       "      <th>Dystopia (1.92) + residual</th>\n",
       "      <th>Explained by: GDP per capita</th>\n",
       "      <th>Explained by: Social support</th>\n",
       "      <th>Explained by: Healthy life expectancy</th>\n",
       "      <th>Explained by: Freedom to make life choices</th>\n",
       "      <th>Explained by: Generosity</th>\n",
       "      <th>Explained by: Perceptions of corruption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Finland</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>1.305</td>\n",
       "      <td>1.592</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Norway</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>2.383</td>\n",
       "      <td>1.456</td>\n",
       "      <td>1.582</td>\n",
       "      <td>0.861</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>7.398</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Country  Happiness score  Whisker-high  Whisker-low  \\\n",
       "0      Finland            7.632         7.695        7.569   \n",
       "1       Norway            7.594         7.657        7.530   \n",
       "2      Denmark            7.555         7.623        7.487   \n",
       "3      Iceland            7.495         7.593        7.398   \n",
       "4  Switzerland            7.487         7.570        7.405   \n",
       "\n",
       "   Dystopia (1.92) + residual  Explained by: GDP per capita  \\\n",
       "0                       2.595                         1.305   \n",
       "1                       2.383                         1.456   \n",
       "2                       2.370                         1.351   \n",
       "3                       2.426                         1.343   \n",
       "4                       2.320                         1.420   \n",
       "\n",
       "   Explained by: Social support  Explained by: Healthy life expectancy  \\\n",
       "0                         1.592                                  0.874   \n",
       "1                         1.582                                  0.861   \n",
       "2                         1.590                                  0.868   \n",
       "3                         1.644                                  0.914   \n",
       "4                         1.549                                  0.927   \n",
       "\n",
       "   Explained by: Freedom to make life choices  Explained by: Generosity  \\\n",
       "0                                       0.681                     0.192   \n",
       "1                                       0.686                     0.286   \n",
       "2                                       0.683                     0.284   \n",
       "3                                       0.677                     0.353   \n",
       "4                                       0.660                     0.256   \n",
       "\n",
       "   Explained by: Perceptions of corruption  \n",
       "0                                    0.393  \n",
       "1                                    0.340  \n",
       "2                                    0.408  \n",
       "3                                    0.138  \n",
       "4                                    0.357  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2 = pd.read_csv('data2.txt', header=[2], comment = '/')\n",
    "data2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "706aed1d-f9f7-42dd-ae2f-8b4179d76423",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>Happiness score</th>\n",
       "      <th>Whisker-high</th>\n",
       "      <th>Whisker-low</th>\n",
       "      <th>Dystopia (1.92) + residual</th>\n",
       "      <th>Explained by: GDP per capita</th>\n",
       "      <th>Explained by: Social support</th>\n",
       "      <th>Explained by: Healthy life expectancy</th>\n",
       "      <th>Explained by: Freedom to make life choices</th>\n",
       "      <th>Explained by: Generosity</th>\n",
       "      <th>Explained by: Perceptions of corruption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Finland</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>1.305</td>\n",
       "      <td>1.592</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Norway</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>2.383</td>\n",
       "      <td>1.456</td>\n",
       "      <td>1.582</td>\n",
       "      <td>0.861</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>7.398</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Country  Happiness score  Whisker-high  Whisker-low  \\\n",
       "0      Finland            7.632         7.695        7.569   \n",
       "1       Norway            7.594         7.657        7.530   \n",
       "2      Denmark            7.555         7.623        7.487   \n",
       "3      Iceland            7.495         7.593        7.398   \n",
       "4  Switzerland            7.487         7.570        7.405   \n",
       "\n",
       "   Dystopia (1.92) + residual  Explained by: GDP per capita  \\\n",
       "0                       2.595                         1.305   \n",
       "1                       2.383                         1.456   \n",
       "2                       2.370                         1.351   \n",
       "3                       2.426                         1.343   \n",
       "4                       2.320                         1.420   \n",
       "\n",
       "   Explained by: Social support  Explained by: Healthy life expectancy  \\\n",
       "0                         1.592                                  0.874   \n",
       "1                         1.582                                  0.861   \n",
       "2                         1.590                                  0.868   \n",
       "3                         1.644                                  0.914   \n",
       "4                         1.549                                  0.927   \n",
       "\n",
       "   Explained by: Freedom to make life choices  Explained by: Generosity  \\\n",
       "0                                       0.681                     0.192   \n",
       "1                                       0.686                     0.286   \n",
       "2                                       0.683                     0.284   \n",
       "3                                       0.677                     0.353   \n",
       "4                                       0.660                     0.256   \n",
       "\n",
       "   Explained by: Perceptions of corruption  \n",
       "0                                    0.393  \n",
       "1                                    0.340  \n",
       "2                                    0.408  \n",
       "3                                    0.138  \n",
       "4                                    0.357  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data3 = pd.read_csv('data3.txt', sep='\\t', header=2)\n",
    "data3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5dab8c96-9b0c-4241-97f6-aeeb285dfb10",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_62/524974606.py:1: FutureWarning: The prefix argument has been deprecated and will be removed in a future version. Use a list comprehension on the column names in the future.\n",
      "\n",
      "\n",
      "  data4 = pd.read_csv('data4.txt', sep='$', header=None, prefix='x')\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x0</th>\n",
       "      <th>x1</th>\n",
       "      <th>x2</th>\n",
       "      <th>x3</th>\n",
       "      <th>x4</th>\n",
       "      <th>x5</th>\n",
       "      <th>x6</th>\n",
       "      <th>x7</th>\n",
       "      <th>x8</th>\n",
       "      <th>x9</th>\n",
       "      <th>x10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Finland</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>1.305</td>\n",
       "      <td>1.592</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Norway</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>2.383</td>\n",
       "      <td>1.456</td>\n",
       "      <td>1.582</td>\n",
       "      <td>0.861</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>7.398</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            x0     x1     x2     x3     x4     x5     x6     x7     x8     x9  \\\n",
       "0      Finland  7.632  7.695  7.569  2.595  1.305  1.592  0.874  0.681  0.192   \n",
       "1       Norway  7.594  7.657  7.530  2.383  1.456  1.582  0.861  0.686  0.286   \n",
       "2      Denmark  7.555  7.623  7.487  2.370  1.351  1.590  0.868  0.683  0.284   \n",
       "3      Iceland  7.495  7.593  7.398  2.426  1.343  1.644  0.914  0.677  0.353   \n",
       "4  Switzerland  7.487  7.570  7.405  2.320  1.420  1.549  0.927  0.660  0.256   \n",
       "\n",
       "     x10  \n",
       "0  0.393  \n",
       "1  0.340  \n",
       "2  0.408  \n",
       "3  0.138  \n",
       "4  0.357  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data4 = pd.read_csv('data4.txt', sep='$', header=None, prefix='x')\n",
    "data4.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9f833e17-ae9b-4a1c-890d-0112abb98210",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_62/489917631.py:1: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support skipfooter; you can avoid this warning by specifying engine='python'.\n",
      "  data5 = pd.read_csv('data5.csv', skipfooter=2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>Happiness score</th>\n",
       "      <th>Whisker-high</th>\n",
       "      <th>Whisker-low</th>\n",
       "      <th>Dystopia (1.92) + residual</th>\n",
       "      <th>Explained by: GDP per capita</th>\n",
       "      <th>Explained by: Social support</th>\n",
       "      <th>Explained by: Healthy life expectancy</th>\n",
       "      <th>Explained by: Freedom to make life choices</th>\n",
       "      <th>Explained by: Generosity</th>\n",
       "      <th>Explained by: Perceptions of corruption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Finland</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>1.305</td>\n",
       "      <td>1.592</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Norway</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>2.383</td>\n",
       "      <td>1.456</td>\n",
       "      <td>1.582</td>\n",
       "      <td>0.861</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>7.398</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Country  Happiness score  Whisker-high  Whisker-low  \\\n",
       "0      Finland            7.632         7.695        7.569   \n",
       "1       Norway            7.594         7.657        7.530   \n",
       "2      Denmark            7.555         7.623        7.487   \n",
       "3      Iceland            7.495         7.593        7.398   \n",
       "4  Switzerland            7.487         7.570        7.405   \n",
       "\n",
       "   Dystopia (1.92) + residual  Explained by: GDP per capita  \\\n",
       "0                       2.595                         1.305   \n",
       "1                       2.383                         1.456   \n",
       "2                       2.370                         1.351   \n",
       "3                       2.426                         1.343   \n",
       "4                       2.320                         1.420   \n",
       "\n",
       "   Explained by: Social support  Explained by: Healthy life expectancy  \\\n",
       "0                         1.592                                  0.874   \n",
       "1                         1.582                                  0.861   \n",
       "2                         1.590                                  0.868   \n",
       "3                         1.644                                  0.914   \n",
       "4                         1.549                                  0.927   \n",
       "\n",
       "   Explained by: Freedom to make life choices  Explained by: Generosity  \\\n",
       "0                                       0.681                     0.192   \n",
       "1                                       0.686                     0.286   \n",
       "2                                       0.683                     0.284   \n",
       "3                                       0.677                     0.353   \n",
       "4                                       0.660                     0.256   \n",
       "\n",
       "   Explained by: Perceptions of corruption  \n",
       "0                                    0.393  \n",
       "1                                    0.340  \n",
       "2                                    0.408  \n",
       "3                                    0.138  \n",
       "4                                    0.357  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data5 = pd.read_csv('data5.csv', skipfooter=2)\n",
    "data5.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9f2ed1d8-4530-4146-9c35-ff2e03800008",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>Happiness score</th>\n",
       "      <th>Whisker-high</th>\n",
       "      <th>Whisker-low</th>\n",
       "      <th>Dystopia (1.92) + residual</th>\n",
       "      <th>Explained by: GDP per capita</th>\n",
       "      <th>Explained by: Social support</th>\n",
       "      <th>Explained by: Healthy life expectancy</th>\n",
       "      <th>Explained by: Freedom to make life choices</th>\n",
       "      <th>Explained by: Generosity</th>\n",
       "      <th>Explained by: Perceptions of corruption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Finland</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>999.000</td>\n",
       "      <td>999.000</td>\n",
       "      <td>999.000</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Norway</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>999.000</td>\n",
       "      <td>999.000</td>\n",
       "      <td>1.582</td>\n",
       "      <td>999.000</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>999.000</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>999.000</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>999.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Country  Happiness score  Whisker-high  Whisker-low  \\\n",
       "0      Finland            7.632         7.695        7.569   \n",
       "1       Norway            7.594         7.657        7.530   \n",
       "2      Denmark            7.555         7.623        7.487   \n",
       "3      Iceland            7.495         7.593      999.000   \n",
       "4  Switzerland            7.487         7.570        7.405   \n",
       "\n",
       "   Dystopia (1.92) + residual  Explained by: GDP per capita  \\\n",
       "0                       2.595                       999.000   \n",
       "1                     999.000                       999.000   \n",
       "2                       2.370                         1.351   \n",
       "3                       2.426                         1.343   \n",
       "4                       2.320                         1.420   \n",
       "\n",
       "   Explained by: Social support  Explained by: Healthy life expectancy  \\\n",
       "0                       999.000                                999.000   \n",
       "1                         1.582                                999.000   \n",
       "2                         1.590                                999.000   \n",
       "3                         1.644                                  0.914   \n",
       "4                         1.549                                  0.927   \n",
       "\n",
       "   Explained by: Freedom to make life choices  Explained by: Generosity  \\\n",
       "0                                       0.681                     0.192   \n",
       "1                                       0.686                     0.286   \n",
       "2                                       0.683                     0.284   \n",
       "3                                       0.677                     0.353   \n",
       "4                                       0.660                     0.256   \n",
       "\n",
       "   Explained by: Perceptions of corruption  \n",
       "0                                    0.393  \n",
       "1                                    0.340  \n",
       "2                                    0.408  \n",
       "3                                  999.000  \n",
       "4                                    0.357  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data6 = pd.read_csv('data6.dat')\n",
    "data6.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "836a8a5b-6061-48f4-bb2e-f4b7db7973bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: openpyxl in /usr/local/lib/python3.10/site-packages (3.0.10)\n",
      "Requirement already satisfied: et-xmlfile in /usr/local/lib/python3.10/site-packages (from openpyxl) (1.1.0)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>Happiness score</th>\n",
       "      <th>Whisker-high</th>\n",
       "      <th>Whisker-low</th>\n",
       "      <th>Dystopia (1.92) + residual</th>\n",
       "      <th>Explained by: GDP per capita</th>\n",
       "      <th>Explained by: Social support</th>\n",
       "      <th>Explained by: Healthy life expectancy</th>\n",
       "      <th>Explained by: Freedom to make life choices</th>\n",
       "      <th>Explained by: Generosity</th>\n",
       "      <th>Explained by: Perceptions of corruption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Finland</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>1.305</td>\n",
       "      <td>1.592</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Norway</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>2.383</td>\n",
       "      <td>1.456</td>\n",
       "      <td>1.582</td>\n",
       "      <td>0.861</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>7.398</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Country  Happiness score  Whisker-high  Whisker-low  \\\n",
       "0      Finland            7.632         7.695        7.569   \n",
       "1       Norway            7.594         7.657        7.530   \n",
       "2      Denmark            7.555         7.623        7.487   \n",
       "3      Iceland            7.495         7.593        7.398   \n",
       "4  Switzerland            7.487         7.570        7.405   \n",
       "\n",
       "   Dystopia (1.92) + residual  Explained by: GDP per capita  \\\n",
       "0                       2.595                         1.305   \n",
       "1                       2.383                         1.456   \n",
       "2                       2.370                         1.351   \n",
       "3                       2.426                         1.343   \n",
       "4                       2.320                         1.420   \n",
       "\n",
       "   Explained by: Social support  Explained by: Healthy life expectancy  \\\n",
       "0                         1.592                                  0.874   \n",
       "1                         1.582                                  0.861   \n",
       "2                         1.590                                  0.868   \n",
       "3                         1.644                                  0.914   \n",
       "4                         1.549                                  0.927   \n",
       "\n",
       "   Explained by: Freedom to make life choices  Explained by: Generosity  \\\n",
       "0                                       0.681                     0.192   \n",
       "1                                       0.686                     0.286   \n",
       "2                                       0.683                     0.284   \n",
       "3                                       0.677                     0.353   \n",
       "4                                       0.660                     0.256   \n",
       "\n",
       "   Explained by: Perceptions of corruption  \n",
       "0                                    0.393  \n",
       "1                                    0.340  \n",
       "2                                    0.408  \n",
       "3                                    0.138  \n",
       "4                                    0.357  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!pip install openpyxl\n",
    "data7 = pd.read_excel('data7.xlsx', sheet_name='Data')\n",
    "data7.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "211316a1-2047-43af-bb53-97ee15f81ef3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>happinessscore</th>\n",
       "      <th>whiskerhigh</th>\n",
       "      <th>whiskerlow</th>\n",
       "      <th>dystopia192residual</th>\n",
       "      <th>explainedbygdppercapita</th>\n",
       "      <th>explainedbysocialsupport</th>\n",
       "      <th>explainedbyhealthylifeexpectancy</th>\n",
       "      <th>explainedbyfreedomtomakelifechoi</th>\n",
       "      <th>explainedbygenerosity</th>\n",
       "      <th>explainedbyperceptionsofcorrupti</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Finland</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>1.305</td>\n",
       "      <td>1.592</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Norway</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>2.383</td>\n",
       "      <td>1.456</td>\n",
       "      <td>1.582</td>\n",
       "      <td>0.861</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>7.398</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  happinessscore  whiskerhigh  whiskerlow  dystopia192residual  \\\n",
       "0      Finland           7.632        7.695       7.569                2.595   \n",
       "1       Norway           7.594        7.657       7.530                2.383   \n",
       "2      Denmark           7.555        7.623       7.487                2.370   \n",
       "3      Iceland           7.495        7.593       7.398                2.426   \n",
       "4  Switzerland           7.487        7.570       7.405                2.320   \n",
       "\n",
       "   explainedbygdppercapita  explainedbysocialsupport  \\\n",
       "0                    1.305                     1.592   \n",
       "1                    1.456                     1.582   \n",
       "2                    1.351                     1.590   \n",
       "3                    1.343                     1.644   \n",
       "4                    1.420                     1.549   \n",
       "\n",
       "   explainedbyhealthylifeexpectancy  explainedbyfreedomtomakelifechoi  \\\n",
       "0                             0.874                             0.681   \n",
       "1                             0.861                             0.686   \n",
       "2                             0.868                             0.683   \n",
       "3                             0.914                             0.677   \n",
       "4                             0.927                             0.660   \n",
       "\n",
       "   explainedbygenerosity  explainedbyperceptionsofcorrupti  \n",
       "0                  0.192                             0.393  \n",
       "1                  0.286                             0.340  \n",
       "2                  0.284                             0.408  \n",
       "3                  0.353                             0.138  \n",
       "4                  0.256                             0.357  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data8 = pd.read_stata('data8.dta')\n",
    "data8.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bd6814c3-fc80-48a3-b29e-76a1e403b283",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pyreadstat in /usr/local/lib/python3.10/site-packages (1.1.9)\n",
      "Requirement already satisfied: pandas>=1.2.0 in /usr/local/lib/python3.10/site-packages (from pyreadstat) (1.4.4)\n",
      "Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/site-packages (from pandas>=1.2.0->pyreadstat) (1.23.3)\n",
      "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/site-packages (from pandas>=1.2.0->pyreadstat) (2022.4)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/site-packages (from pandas>=1.2.0->pyreadstat) (2.8.2)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/site-packages (from python-dateutil>=2.8.1->pandas>=1.2.0->pyreadstat) (1.16.0)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>happiness</th>\n",
       "      <th>whiskerhigh</th>\n",
       "      <th>whiskerlow</th>\n",
       "      <th>dystopia</th>\n",
       "      <th>gdpPC</th>\n",
       "      <th>socsupport</th>\n",
       "      <th>lifeexp</th>\n",
       "      <th>lifechoice</th>\n",
       "      <th>generous</th>\n",
       "      <th>corrupt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Finland</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>1.305</td>\n",
       "      <td>1.592</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Norway</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>2.383</td>\n",
       "      <td>1.456</td>\n",
       "      <td>1.582</td>\n",
       "      <td>0.861</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>7.398</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  happiness  whiskerhigh  whiskerlow  dystopia  gdpPC  \\\n",
       "0      Finland      7.632        7.695       7.569     2.595  1.305   \n",
       "1       Norway      7.594        7.657       7.530     2.383  1.456   \n",
       "2      Denmark      7.555        7.623       7.487     2.370  1.351   \n",
       "3      Iceland      7.495        7.593       7.398     2.426  1.343   \n",
       "4  Switzerland      7.487        7.570       7.405     2.320  1.420   \n",
       "\n",
       "   socsupport  lifeexp  lifechoice  generous  corrupt  \n",
       "0       1.592    0.874       0.681     0.192    0.393  \n",
       "1       1.582    0.861       0.686     0.286    0.340  \n",
       "2       1.590    0.868       0.683     0.284    0.408  \n",
       "3       1.644    0.914       0.677     0.353    0.138  \n",
       "4       1.549    0.927       0.660     0.256    0.357  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!pip install pyreadstat\n",
    "data9 = pd.read_spss('data9.sav')\n",
    "data9.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "5d8300fa-a991-4759-862e-c17097185b6c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>COUNTRY</th>\n",
       "      <th>HAPPINES</th>\n",
       "      <th>WHISKERH</th>\n",
       "      <th>WHISKERL</th>\n",
       "      <th>DYSTOPIA</th>\n",
       "      <th>EXPLAINE</th>\n",
       "      <th>EXPLAIN2</th>\n",
       "      <th>EXPLAIN3</th>\n",
       "      <th>EXPLAIN4</th>\n",
       "      <th>EXPLAIN5</th>\n",
       "      <th>EXPLAIN6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>b'Finland'</td>\n",
       "      <td>7.632</td>\n",
       "      <td>7.695</td>\n",
       "      <td>7.569</td>\n",
       "      <td>2.595</td>\n",
       "      <td>1.305</td>\n",
       "      <td>1.592</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b'Norway'</td>\n",
       "      <td>7.594</td>\n",
       "      <td>7.657</td>\n",
       "      <td>7.530</td>\n",
       "      <td>2.383</td>\n",
       "      <td>1.456</td>\n",
       "      <td>1.582</td>\n",
       "      <td>0.861</td>\n",
       "      <td>0.686</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b'Denmark'</td>\n",
       "      <td>7.555</td>\n",
       "      <td>7.623</td>\n",
       "      <td>7.487</td>\n",
       "      <td>2.370</td>\n",
       "      <td>1.351</td>\n",
       "      <td>1.590</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.683</td>\n",
       "      <td>0.284</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b'Iceland'</td>\n",
       "      <td>7.495</td>\n",
       "      <td>7.593</td>\n",
       "      <td>7.398</td>\n",
       "      <td>2.426</td>\n",
       "      <td>1.343</td>\n",
       "      <td>1.644</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.353</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b'Switzerland'</td>\n",
       "      <td>7.487</td>\n",
       "      <td>7.570</td>\n",
       "      <td>7.405</td>\n",
       "      <td>2.320</td>\n",
       "      <td>1.420</td>\n",
       "      <td>1.549</td>\n",
       "      <td>0.927</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          COUNTRY  HAPPINES  WHISKERH  WHISKERL  DYSTOPIA  EXPLAINE  EXPLAIN2  \\\n",
       "0      b'Finland'     7.632     7.695     7.569     2.595     1.305     1.592   \n",
       "1       b'Norway'     7.594     7.657     7.530     2.383     1.456     1.582   \n",
       "2      b'Denmark'     7.555     7.623     7.487     2.370     1.351     1.590   \n",
       "3      b'Iceland'     7.495     7.593     7.398     2.426     1.343     1.644   \n",
       "4  b'Switzerland'     7.487     7.570     7.405     2.320     1.420     1.549   \n",
       "\n",
       "   EXPLAIN3  EXPLAIN4  EXPLAIN5  EXPLAIN6  \n",
       "0     0.874     0.681     0.192     0.393  \n",
       "1     0.861     0.686     0.286     0.340  \n",
       "2     0.868     0.683     0.284     0.408  \n",
       "3     0.914     0.677     0.353     0.138  \n",
       "4     0.927     0.660     0.256     0.357  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data10 = pd.read_sas('data10.xpt')\n",
    "data10.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73605216-6eee-4bff-91a9-38a10d8ec243",
   "metadata": {},
   "source": [
    "(3) For this problem, you will be accessing the open APIs maintained by NASA. All of the API endpoints share a credentialing system with the same API key. First, register for an API key here: https://api.nasa.gov/. Once you have the key, save it in your .env file. \n",
    "\n",
    "Then click on Browse APIs and find the information for the Asteroids - NeoWs API. This API reports data on \"near Earth objects\" in space, such as asteroids. It reports the size of the objects, their speed, direction, and distance to Earth. It even codes whether or not NASA considers the object to be potentially hazardous to Earth.\n",
    "\n",
    "Write the code to access this API and generate a single data frame with all of the near Earth objects reported by NASA over the last 7 days.\n",
    "\n",
    "To receive full credit for this problem, make sure you\n",
    "\n",
    "* Supply an accurate user-agent in the headers along with your email address in the 'From' field\n",
    "\n",
    "* Provide your NASA API key in the way the documentation instructs\n",
    "\n",
    "* Use `pd.json_normalize()` and `pd.concat()` to extract and combine several data frames from the JSON output.\n",
    "\n",
    "Note: in your output, the `close_approach_data` field will still be in a dictionary within a list in the final data frame. That's okay for this problem. If you want to extract those data and store them as additional columns, please do: I recommend using list comprehensions as follows:\n",
    "```\n",
    "df['close_approach_date'] = [d[0][''close_approach_date''] for d in df['close_approach_data']]\n",
    "```\n",
    "[3 points]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "07507a00-cf55-4736-9b78-f2a78dc887de",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(oldpath)\n",
    "nasa_token = os.environ['nasa_token']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4d715ef4-b27d-4e9a-be6a-77943a322376",
   "metadata": {},
   "outputs": [],
   "source": [
    "useragent_url = 'https://httpbin.org/user-agent'\n",
    "r = requests.get(useragent_url)\n",
    "useragent = json.loads(r.text)['user-agent']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "964f5fe2-221e-48eb-a2a5-dd5792f23a5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {'User-Agent' : useragent,\n",
    "          'From': 'bwl5cd@virginia.edu'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "75ad421e-0551-41a5-bb65-912339882c88",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "root = 'https://api.nasa.gov'\n",
    "start_date = '2022-10-6'\n",
    "asteroids_endpoint = '/neo/rest/v1/feed?start_date={start_date}&api_key={api_key}'.format(start_date=start_date, \n",
    "                                                                                          api_key=nasa_token)\n",
    "r = requests.get(root + asteroids_endpoint,\n",
    "                headers = headers)\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1297de3b-dd9f-46f9-93ad-09c55bf8e21d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>neo_reference_id</th>\n",
       "      <th>name</th>\n",
       "      <th>nasa_jpl_url</th>\n",
       "      <th>absolute_magnitude_h</th>\n",
       "      <th>is_potentially_hazardous_asteroid</th>\n",
       "      <th>close_approach_data</th>\n",
       "      <th>is_sentry_object</th>\n",
       "      <th>links.self</th>\n",
       "      <th>estimated_diameter.kilometers.estimated_diameter_min</th>\n",
       "      <th>estimated_diameter.kilometers.estimated_diameter_max</th>\n",
       "      <th>estimated_diameter.meters.estimated_diameter_min</th>\n",
       "      <th>estimated_diameter.meters.estimated_diameter_max</th>\n",
       "      <th>estimated_diameter.miles.estimated_diameter_min</th>\n",
       "      <th>estimated_diameter.miles.estimated_diameter_max</th>\n",
       "      <th>estimated_diameter.feet.estimated_diameter_min</th>\n",
       "      <th>estimated_diameter.feet.estimated_diameter_max</th>\n",
       "      <th>close_approach_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2163692</td>\n",
       "      <td>2163692</td>\n",
       "      <td>163692 (2003 CY18)</td>\n",
       "      <td>http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2163692</td>\n",
       "      <td>18.26</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'close_approach_date': '2022-10-06', 'close_...</td>\n",
       "      <td>False</td>\n",
       "      <td>http://api.nasa.gov/neo/rest/v1/neo/2163692?ap...</td>\n",
       "      <td>0.592318</td>\n",
       "      <td>1.324463</td>\n",
       "      <td>592.318063</td>\n",
       "      <td>1324.463452</td>\n",
       "      <td>0.368049</td>\n",
       "      <td>0.822983</td>\n",
       "      <td>1943.300793</td>\n",
       "      <td>4345.352673</td>\n",
       "      <td>2022-10-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2516428</td>\n",
       "      <td>2516428</td>\n",
       "      <td>516428 (2003 UR12)</td>\n",
       "      <td>http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2516428</td>\n",
       "      <td>17.90</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'close_approach_date': '2022-10-06', 'close_...</td>\n",
       "      <td>False</td>\n",
       "      <td>http://api.nasa.gov/neo/rest/v1/neo/2516428?ap...</td>\n",
       "      <td>0.699125</td>\n",
       "      <td>1.563292</td>\n",
       "      <td>699.125232</td>\n",
       "      <td>1563.291544</td>\n",
       "      <td>0.434416</td>\n",
       "      <td>0.971384</td>\n",
       "      <td>2293.718027</td>\n",
       "      <td>5128.909430</td>\n",
       "      <td>2022-10-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3291224</td>\n",
       "      <td>3291224</td>\n",
       "      <td>(2005 SP9)</td>\n",
       "      <td>http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3291224</td>\n",
       "      <td>21.64</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'close_approach_date': '2022-10-06', 'close_...</td>\n",
       "      <td>False</td>\n",
       "      <td>http://api.nasa.gov/neo/rest/v1/neo/3291224?ap...</td>\n",
       "      <td>0.124898</td>\n",
       "      <td>0.279280</td>\n",
       "      <td>124.897854</td>\n",
       "      <td>279.280092</td>\n",
       "      <td>0.077608</td>\n",
       "      <td>0.173537</td>\n",
       "      <td>409.769876</td>\n",
       "      <td>916.273297</td>\n",
       "      <td>2022-10-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3826795</td>\n",
       "      <td>3826795</td>\n",
       "      <td>(2018 PZ21)</td>\n",
       "      <td>http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3826795</td>\n",
       "      <td>26.70</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'close_approach_date': '2022-10-06', 'close_...</td>\n",
       "      <td>False</td>\n",
       "      <td>http://api.nasa.gov/neo/rest/v1/neo/3826795?ap...</td>\n",
       "      <td>0.012149</td>\n",
       "      <td>0.027167</td>\n",
       "      <td>12.149404</td>\n",
       "      <td>27.166893</td>\n",
       "      <td>0.007549</td>\n",
       "      <td>0.016881</td>\n",
       "      <td>39.860251</td>\n",
       "      <td>89.130231</td>\n",
       "      <td>2022-10-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3831613</td>\n",
       "      <td>3831613</td>\n",
       "      <td>(2018 TL3)</td>\n",
       "      <td>http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3831613</td>\n",
       "      <td>24.40</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'close_approach_date': '2022-10-06', 'close_...</td>\n",
       "      <td>False</td>\n",
       "      <td>http://api.nasa.gov/neo/rest/v1/neo/3831613?ap...</td>\n",
       "      <td>0.035039</td>\n",
       "      <td>0.078350</td>\n",
       "      <td>35.039264</td>\n",
       "      <td>78.350176</td>\n",
       "      <td>0.021772</td>\n",
       "      <td>0.048685</td>\n",
       "      <td>114.958219</td>\n",
       "      <td>257.054393</td>\n",
       "      <td>2022-10-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>54075318</td>\n",
       "      <td>54075318</td>\n",
       "      <td>(2020 US1)</td>\n",
       "      <td>http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54075318</td>\n",
       "      <td>25.09</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'close_approach_date': '2022-10-12', 'close_...</td>\n",
       "      <td>False</td>\n",
       "      <td>http://api.nasa.gov/neo/rest/v1/neo/54075318?a...</td>\n",
       "      <td>0.025501</td>\n",
       "      <td>0.057022</td>\n",
       "      <td>25.500869</td>\n",
       "      <td>57.021676</td>\n",
       "      <td>0.015846</td>\n",
       "      <td>0.035432</td>\n",
       "      <td>83.664270</td>\n",
       "      <td>187.078996</td>\n",
       "      <td>2022-10-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>2395289</td>\n",
       "      <td>2395289</td>\n",
       "      <td>395289 (2011 BJ2)</td>\n",
       "      <td>http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2395289</td>\n",
       "      <td>18.30</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'close_approach_date': '2022-10-13', 'close_...</td>\n",
       "      <td>False</td>\n",
       "      <td>http://api.nasa.gov/neo/rest/v1/neo/2395289?ap...</td>\n",
       "      <td>0.581507</td>\n",
       "      <td>1.300289</td>\n",
       "      <td>581.507040</td>\n",
       "      <td>1300.289270</td>\n",
       "      <td>0.361332</td>\n",
       "      <td>0.807962</td>\n",
       "      <td>1907.831556</td>\n",
       "      <td>4266.041049</td>\n",
       "      <td>2022-10-13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>2519354</td>\n",
       "      <td>2519354</td>\n",
       "      <td>519354 (2011 KR12)</td>\n",
       "      <td>http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2519354</td>\n",
       "      <td>21.51</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'close_approach_date': '2022-10-13', 'close_...</td>\n",
       "      <td>False</td>\n",
       "      <td>http://api.nasa.gov/neo/rest/v1/neo/2519354?ap...</td>\n",
       "      <td>0.132603</td>\n",
       "      <td>0.296510</td>\n",
       "      <td>132.603497</td>\n",
       "      <td>296.510433</td>\n",
       "      <td>0.082396</td>\n",
       "      <td>0.184243</td>\n",
       "      <td>435.050856</td>\n",
       "      <td>972.803288</td>\n",
       "      <td>2022-10-13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>54104724</td>\n",
       "      <td>54104724</td>\n",
       "      <td>(2021 AE2)</td>\n",
       "      <td>http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54104724</td>\n",
       "      <td>24.87</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'close_approach_date': '2022-10-13', 'close_...</td>\n",
       "      <td>False</td>\n",
       "      <td>http://api.nasa.gov/neo/rest/v1/neo/54104724?a...</td>\n",
       "      <td>0.028220</td>\n",
       "      <td>0.063102</td>\n",
       "      <td>28.219868</td>\n",
       "      <td>63.101543</td>\n",
       "      <td>0.017535</td>\n",
       "      <td>0.039209</td>\n",
       "      <td>92.584871</td>\n",
       "      <td>207.026066</td>\n",
       "      <td>2022-10-13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>54145785</td>\n",
       "      <td>54145785</td>\n",
       "      <td>(2021 KC)</td>\n",
       "      <td>http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54145785</td>\n",
       "      <td>23.47</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'close_approach_date': '2022-10-13', 'close_...</td>\n",
       "      <td>False</td>\n",
       "      <td>http://api.nasa.gov/neo/rest/v1/neo/54145785?a...</td>\n",
       "      <td>0.053772</td>\n",
       "      <td>0.120238</td>\n",
       "      <td>53.771850</td>\n",
       "      <td>120.237511</td>\n",
       "      <td>0.033412</td>\n",
       "      <td>0.074712</td>\n",
       "      <td>176.416836</td>\n",
       "      <td>394.480037</td>\n",
       "      <td>2022-10-13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>83 rows × 18 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          id neo_reference_id                name  \\\n",
       "0    2163692          2163692  163692 (2003 CY18)   \n",
       "1    2516428          2516428  516428 (2003 UR12)   \n",
       "2    3291224          3291224          (2005 SP9)   \n",
       "3    3826795          3826795         (2018 PZ21)   \n",
       "4    3831613          3831613          (2018 TL3)   \n",
       "..       ...              ...                 ...   \n",
       "78  54075318         54075318          (2020 US1)   \n",
       "79   2395289          2395289   395289 (2011 BJ2)   \n",
       "80   2519354          2519354  519354 (2011 KR12)   \n",
       "81  54104724         54104724          (2021 AE2)   \n",
       "82  54145785         54145785           (2021 KC)   \n",
       "\n",
       "                                      nasa_jpl_url  absolute_magnitude_h  \\\n",
       "0    http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2163692                 18.26   \n",
       "1    http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2516428                 17.90   \n",
       "2    http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3291224                 21.64   \n",
       "3    http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3826795                 26.70   \n",
       "4    http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3831613                 24.40   \n",
       "..                                             ...                   ...   \n",
       "78  http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54075318                 25.09   \n",
       "79   http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2395289                 18.30   \n",
       "80   http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2519354                 21.51   \n",
       "81  http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54104724                 24.87   \n",
       "82  http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54145785                 23.47   \n",
       "\n",
       "    is_potentially_hazardous_asteroid  \\\n",
       "0                               False   \n",
       "1                               False   \n",
       "2                               False   \n",
       "3                               False   \n",
       "4                               False   \n",
       "..                                ...   \n",
       "78                              False   \n",
       "79                              False   \n",
       "80                              False   \n",
       "81                              False   \n",
       "82                              False   \n",
       "\n",
       "                                  close_approach_data  is_sentry_object  \\\n",
       "0   [{'close_approach_date': '2022-10-06', 'close_...             False   \n",
       "1   [{'close_approach_date': '2022-10-06', 'close_...             False   \n",
       "2   [{'close_approach_date': '2022-10-06', 'close_...             False   \n",
       "3   [{'close_approach_date': '2022-10-06', 'close_...             False   \n",
       "4   [{'close_approach_date': '2022-10-06', 'close_...             False   \n",
       "..                                                ...               ...   \n",
       "78  [{'close_approach_date': '2022-10-12', 'close_...             False   \n",
       "79  [{'close_approach_date': '2022-10-13', 'close_...             False   \n",
       "80  [{'close_approach_date': '2022-10-13', 'close_...             False   \n",
       "81  [{'close_approach_date': '2022-10-13', 'close_...             False   \n",
       "82  [{'close_approach_date': '2022-10-13', 'close_...             False   \n",
       "\n",
       "                                           links.self  \\\n",
       "0   http://api.nasa.gov/neo/rest/v1/neo/2163692?ap...   \n",
       "1   http://api.nasa.gov/neo/rest/v1/neo/2516428?ap...   \n",
       "2   http://api.nasa.gov/neo/rest/v1/neo/3291224?ap...   \n",
       "3   http://api.nasa.gov/neo/rest/v1/neo/3826795?ap...   \n",
       "4   http://api.nasa.gov/neo/rest/v1/neo/3831613?ap...   \n",
       "..                                                ...   \n",
       "78  http://api.nasa.gov/neo/rest/v1/neo/54075318?a...   \n",
       "79  http://api.nasa.gov/neo/rest/v1/neo/2395289?ap...   \n",
       "80  http://api.nasa.gov/neo/rest/v1/neo/2519354?ap...   \n",
       "81  http://api.nasa.gov/neo/rest/v1/neo/54104724?a...   \n",
       "82  http://api.nasa.gov/neo/rest/v1/neo/54145785?a...   \n",
       "\n",
       "    estimated_diameter.kilometers.estimated_diameter_min  \\\n",
       "0                                            0.592318      \n",
       "1                                            0.699125      \n",
       "2                                            0.124898      \n",
       "3                                            0.012149      \n",
       "4                                            0.035039      \n",
       "..                                                ...      \n",
       "78                                           0.025501      \n",
       "79                                           0.581507      \n",
       "80                                           0.132603      \n",
       "81                                           0.028220      \n",
       "82                                           0.053772      \n",
       "\n",
       "    estimated_diameter.kilometers.estimated_diameter_max  \\\n",
       "0                                            1.324463      \n",
       "1                                            1.563292      \n",
       "2                                            0.279280      \n",
       "3                                            0.027167      \n",
       "4                                            0.078350      \n",
       "..                                                ...      \n",
       "78                                           0.057022      \n",
       "79                                           1.300289      \n",
       "80                                           0.296510      \n",
       "81                                           0.063102      \n",
       "82                                           0.120238      \n",
       "\n",
       "    estimated_diameter.meters.estimated_diameter_min  \\\n",
       "0                                         592.318063   \n",
       "1                                         699.125232   \n",
       "2                                         124.897854   \n",
       "3                                          12.149404   \n",
       "4                                          35.039264   \n",
       "..                                               ...   \n",
       "78                                         25.500869   \n",
       "79                                        581.507040   \n",
       "80                                        132.603497   \n",
       "81                                         28.219868   \n",
       "82                                         53.771850   \n",
       "\n",
       "    estimated_diameter.meters.estimated_diameter_max  \\\n",
       "0                                        1324.463452   \n",
       "1                                        1563.291544   \n",
       "2                                         279.280092   \n",
       "3                                          27.166893   \n",
       "4                                          78.350176   \n",
       "..                                               ...   \n",
       "78                                         57.021676   \n",
       "79                                       1300.289270   \n",
       "80                                        296.510433   \n",
       "81                                         63.101543   \n",
       "82                                        120.237511   \n",
       "\n",
       "    estimated_diameter.miles.estimated_diameter_min  \\\n",
       "0                                          0.368049   \n",
       "1                                          0.434416   \n",
       "2                                          0.077608   \n",
       "3                                          0.007549   \n",
       "4                                          0.021772   \n",
       "..                                              ...   \n",
       "78                                         0.015846   \n",
       "79                                         0.361332   \n",
       "80                                         0.082396   \n",
       "81                                         0.017535   \n",
       "82                                         0.033412   \n",
       "\n",
       "    estimated_diameter.miles.estimated_diameter_max  \\\n",
       "0                                          0.822983   \n",
       "1                                          0.971384   \n",
       "2                                          0.173537   \n",
       "3                                          0.016881   \n",
       "4                                          0.048685   \n",
       "..                                              ...   \n",
       "78                                         0.035432   \n",
       "79                                         0.807962   \n",
       "80                                         0.184243   \n",
       "81                                         0.039209   \n",
       "82                                         0.074712   \n",
       "\n",
       "    estimated_diameter.feet.estimated_diameter_min  \\\n",
       "0                                      1943.300793   \n",
       "1                                      2293.718027   \n",
       "2                                       409.769876   \n",
       "3                                        39.860251   \n",
       "4                                       114.958219   \n",
       "..                                             ...   \n",
       "78                                       83.664270   \n",
       "79                                     1907.831556   \n",
       "80                                      435.050856   \n",
       "81                                       92.584871   \n",
       "82                                      176.416836   \n",
       "\n",
       "    estimated_diameter.feet.estimated_diameter_max close_approach_date  \n",
       "0                                      4345.352673          2022-10-06  \n",
       "1                                      5128.909430          2022-10-06  \n",
       "2                                       916.273297          2022-10-06  \n",
       "3                                        89.130231          2022-10-06  \n",
       "4                                       257.054393          2022-10-06  \n",
       "..                                             ...                 ...  \n",
       "78                                      187.078996          2022-10-12  \n",
       "79                                     4266.041049          2022-10-13  \n",
       "80                                      972.803288          2022-10-13  \n",
       "81                                      207.026066          2022-10-13  \n",
       "82                                      394.480037          2022-10-13  \n",
       "\n",
       "[83 rows x 18 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myjson = json.loads(r.text)\n",
    "asteroids06 = pd.json_normalize(myjson, record_path = ['near_earth_objects', '2022-10-06'])\n",
    "asteroids07 = pd.json_normalize(myjson, record_path = ['near_earth_objects', '2022-10-07'])\n",
    "asteroids08 = pd.json_normalize(myjson, record_path = ['near_earth_objects', '2022-10-08'])\n",
    "asteroids09 = pd.json_normalize(myjson, record_path = ['near_earth_objects', '2022-10-09'])\n",
    "asteroids10 = pd.json_normalize(myjson, record_path = ['near_earth_objects', '2022-10-10'])\n",
    "asteroids11 = pd.json_normalize(myjson, record_path = ['near_earth_objects', '2022-10-11'])\n",
    "asteroids12 = pd.json_normalize(myjson, record_path = ['near_earth_objects', '2022-10-12'])\n",
    "asteroids13 = pd.json_normalize(myjson, record_path = ['near_earth_objects', '2022-10-13'])\n",
    "asteroids = pd.concat([asteroids06, asteroids07, asteroids08, asteroids09, \n",
    "                      asteroids10, asteroids11, asteroids12, asteroids13], ignore_index=True)\n",
    "asteroids['close_approach_date'] = [d[0]['close_approach_date'] for d in asteroids['close_approach_data']]\n",
    "asteroids"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee77ca9f-55d4-48e5-b319-d9490b70c10a",
   "metadata": {},
   "source": [
    "(4) Read the following blog post about how to read a robots.txt file: https://yoast.com/ultimate-guide-robots-txt/ Then look at the robots.txt files for the following websites, and report whether they allow the following kinds of web-scraping. For each question, explain your answer and copy-and-paste the relevant parts of the robots.txt file if necessary:\n",
    "\n",
    "a. Pulling the current listed prices for houses for sale in Charlottesville from zillow.com\n",
    "\n",
    "This is disallowed. The following command dissalows any websraping under the tag \\houses.\n",
    "\n",
    "\"Disallow: /homes/*/?\"\n",
    "\n",
    "b. Scraping current stock prices off of google.com/finance\n",
    "\n",
    "This is allowed. The following allow statement appears in the robots.txt file and would allow scraping with any tag /finance.\n",
    "\n",
    "\"Allow: /finance\"\n",
    "\n",
    "c. Copying the list of twitter accounts that each NBA player follows\n",
    "\n",
    "This is not allowed. The robots.txt file for twitter has a statment that would disallow accessing any tag before the tag /following. The command is repeated for common search engines and even has a comment \"# Every bot that might possibly read and respect this file\" with the following statement under it:\n",
    "\n",
    "\"Disallow: /*/following\"\n",
    "\n",
    "d. The lyrics to Lizzo's Good as Hell from https://genius.com/Lizzo-good-as-hell-lyrics\n",
    "\n",
    "This is allowed. There was no disallow statment corresponding to lyrics, however, there was if you were using Uniscan or Searchie. Both of those webscraping agents are disallowed from crawling anything in the website. \n",
    "\n",
    "[2 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7683f75-e7fe-4a73-8689-7ac297a19e59",
   "metadata": {},
   "source": [
    "(5) In class we accessed the Genius API to find the most popular songs by Lizzo. But the Genius API does not allow us to download the lyrics to any songs. This looks like a job for web scraping. \n",
    "\n",
    "Good As Hell is the 15th track on Lizzo's 2019 album Cuz I Love You. In this problem, you will web-scrape genius.com to bring the lyrics to Good As Hell into Python, find links to the lyrics to the other songs on the Cuz I Love You album, and build a spider to extract all the song lyrics from this album.\n",
    "\n",
    "A few tricks I found useful to complete these tasks:\n",
    "\n",
    "** Trick 1**: A list comprehension is a for-loop across items of a list that places the outputs into another list. The general syntax is\n",
    "```\n",
    "[f(x) for x in list]\n",
    "```\n",
    "where `list` is the list of input values, `x` is a token that stands in for one general element of the input list, and `f(x)` is what we want to do to each element. For example, if this is my list of inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "59ae8d79-45fb-4844-83ee-f485e2c1e6d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = [3,5,8,9,12]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d164987-cddd-4f81-90a2-634a034bd302",
   "metadata": {},
   "source": [
    "and I want to square then subtract 1 from each element, I can use the syntax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "87ef370c-e63f-4911-beff-e740e1c75438",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[8, 24, 63, 80, 143]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(x**2) - 1 for x in inputs]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dce70e2c-a6e2-4da9-a4fe-12d9a097a1d3",
   "metadata": {},
   "source": [
    "For this problem, you will need to use a *filtered* list comprehension, which uses `if` to filter down the list of outputs. For example, if I only want to keep the even input numbers in this list (8 and 12), I can use `if x/2 == round(x/2)` (no decimals when dividing by 2) to narrow the list down:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d80d2459-a7f8-43b1-94ab-0d3444839d49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[63, 143]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(x**2) - 1 for x in inputs if x/2 == round(x/2)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dab915f-bf44-4a3f-8887-c3d5fadff6e2",
   "metadata": {},
   "source": [
    "**Trick 2**: The second trick that will be needed is the `attrs{}` argument of BeautifulSoup's `.find_all()` method. When searching for particular HTML tags, you may want to narrow down the search to only those tags with a particular attribute. For example, if I saved the parsed HTML as `mysoup`, and I want to find all the `a` tags that also have an attribute `rel=\"noopener\"`, I can type\n",
    "```\n",
    "mysoup.find_all('a', attrs{'rel':'noopener'})\n",
    "```\n",
    "If the attribute has no hyphens, and is not a Python reserved variable like `name` or `class`, then a shortcut for the same function is\n",
    "```\n",
    "mysoup.find_all('a', rel='noopener')\n",
    "```\n",
    "but if the attribute name hyphens or is a reserved variable, then the `attrs{}` approach still works.\n",
    "\n",
    "**Trick 3**: Finally, suppose you have one text document in a way in which each paragraph is a separate item of a list. The `.join()` method can combine all these elements into a single string. For example, if I have a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e0bbef60-4d93-4549-a3b3-4d6b66aaae30",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I',\n",
       " 'do',\n",
       " 'my',\n",
       " 'hair',\n",
       " 'toss,',\n",
       " 'check',\n",
       " 'my',\n",
       " 'nails',\n",
       " 'Baby,',\n",
       " 'how',\n",
       " 'you',\n",
       " 'feelin?',\n",
       " 'Feelin',\n",
       " 'good',\n",
       " 'as',\n",
       " 'hell']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = ['I','do','my','hair','toss,', 'check','my','nails','Baby,','how','you','feelin?','Feelin','good','as','hell']\n",
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f995215-4068-4127-a4c2-428a2e2f4d6a",
   "metadata": {},
   "source": [
    "and I want to combine them into a single string in which each word is separated by a space, I type a space in quotes, then apply the `.join(words)` method to it:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "cd6d46e3-738d-4a02-ace6-09bbf068d79e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I do my hair toss, check my nails Baby, how you feelin? Feelin good as hell'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ed3f4e-0d80-43f5-ae2d-9d41e0c4da7b",
   "metadata": {},
   "source": [
    "Using these three tricks, perform the following tasks:\n",
    "\n",
    "(a) Use `requests` and `BeautifulSoup` to scrape the lyrics to Good As Hell from https://genius.com/Lizzo-good-as-hell-lyrics\n",
    "\n",
    "(b) Notice that at towards the bottom of https://genius.com/Lizzo-good-as-hell-lyrics there is a menu containing a track list for the Cuz I Love You album. Each link here takes you to the lyrics for that song. Use `requests` and `BeautifulSoup` to save the list of song lyric URLs in a Python list.\n",
    "\n",
    "(c) Build a spider: first write a function that applies the code you wrote for part (a) to a user-supplied URL; then apply this function to each of the URLs in the list you constructed in part (b).\n",
    "\n",
    "[5 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf48fd67-b263-408e-971f-735e6bc55239",
   "metadata": {},
   "source": [
    "(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "0231158b-c7f5-49ca-bb00-0450a7b2d8de",
   "metadata": {},
   "outputs": [],
   "source": [
    "genius_token = os.environ['genius_token']\n",
    "urltoscrape = 'https://genius.com/Lizzo-good-as-hell-lyrics'\n",
    "#urltoscrape = 'https://genius.com/Lizzo-cuz-i-love-you-lyrics'\n",
    "r = requests.get(urltoscrape,\n",
    "               headers = {'User-Agent': useragent,\n",
    "                         'From': 'bwl5cd@virginia.edu'})\n",
    "my_html = BeautifulSoup(r.text, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "242a7d91-8e53-498d-a6ec-2f1f29ee53fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Chorus]\n",
      "I do my hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "Hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "[Verse 1]\n",
      "Woo, child, tired of the bullshit\n",
      "Go on, dust your shoulders off, keep it moving\n",
      "Yes, Lord, tryna get some new shit\n",
      "In there, swimwear, going-to-the-pool shit\n",
      "Come now, come dry your eyes\n",
      "You know you a star, you can touch the sky\n",
      "I know that it's hard, but you have to try\n",
      "If you need advice, let me simplify\n",
      "[Pre-Chorus]\n",
      "If he don't love you anymore\n",
      "Just walk your fine ass out the door\n",
      "[Chorus]\n",
      "I do my hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "Hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "(Feeling good as hell)\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "[Verse 2]\n",
      "Woo, girl, need to kick off your shoes\n",
      "Gotta take a deep breath, time to focus on you\n",
      "All the big fights, long nights that you been through\n",
      "I got a bottle of tequila I been saving for you\n",
      "Boss up and change your life\n",
      "You can have it all, no sacrifice\n",
      "I know he did you wrong, we can make it right\n",
      "So go and let it all hang out tonight\n",
      "[Pre-Chorus]\n",
      "'Cause he don't love you anymore\n",
      "So walk your fine ass out the door\n",
      "[Chorus]\n",
      "And do your hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "Hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell) (Show it off tonight)\n",
      "Hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell) (And we gon' be alright)\n",
      "Hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "[Pre-Chorus]\n",
      "Mmm, yeah, alright\n",
      "Listen, if he don't love you anymore\n",
      "Then walk your fine ass out the door\n",
      "[Chorus]\n",
      "And do your hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell) (Woo-hoo)\n",
      "Hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell) (Show it off tonight)\n",
      "Hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell) (And we gon' make it right)\n",
      "Hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "(Feelin' good as hell)\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n"
     ]
    }
   ],
   "source": [
    "lyrics = my_html.find_all(class_ = 'Lyrics__Container-sc-1ynbvzw-6 YYrds')\n",
    "lyrics = [search_result.get_text(separator=\"\\n\") for search_result in lyrics]\n",
    "lyrics = '\\n'.join(lyrics)\n",
    "print(lyrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49bb2ec4-289c-43fa-b04d-2ff6f3e8bf96",
   "metadata": {},
   "source": [
    "b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b84c5ee2-7a23-4f0a-9fdd-1eb6da6a9acf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://genius.com/Lizzo-cuz-i-love-you-lyrics',\n",
       " 'https://genius.com/Lizzo-like-a-girl-lyrics',\n",
       " 'https://genius.com/Lizzo-juice-lyrics',\n",
       " 'https://genius.com/Lizzo-soulmate-lyrics',\n",
       " 'https://genius.com/Lizzo-jerome-lyrics',\n",
       " 'https://genius.com/Lizzo-cry-baby-lyrics',\n",
       " 'https://genius.com/Lizzo-tempo-lyrics',\n",
       " 'https://genius.com/Lizzo-exactly-how-i-feel-lyrics',\n",
       " 'https://genius.com/Lizzo-better-in-color-lyrics',\n",
       " 'https://genius.com/Lizzo-heaven-help-me-lyrics',\n",
       " 'https://genius.com/Lizzo-lingerie-lyrics',\n",
       " 'https://genius.com/Lizzo-boys-lyrics',\n",
       " 'https://genius.com/Lizzo-truth-hurts-lyrics',\n",
       " 'https://genius.com/Lizzo-water-me-lyrics',\n",
       " 'https://genius.com/Lizzo-good-as-hell-remix-lyrics']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links = my_html.find_all(class_ = 'AlbumTracklist__Container-sc-123giuo-0 kGJQLs')\n",
    "links = [link.a for link in links[0].contents]\n",
    "links = [link['href'] for link in links if link is not None]\n",
    "links"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6b1da8c-599b-4da1-ba84-6eb7e6d584bd",
   "metadata": {},
   "source": [
    "c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "966cf597-e5bb-4c33-a7b3-021015e2c74e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_lyrics(url, useragent):\n",
    "    r = requests.get(url,\n",
    "                     headers = {'User-Agent': useragent,\n",
    "                                'From': 'bwl5cd@virginia.edu'})\n",
    "    my_html = BeautifulSoup(r.text, 'html.parser')\n",
    "    lyrics = my_html.find_all(class_ = 'Lyrics__Container-sc-1ynbvzw-6 YYrds')\n",
    "    lyrics = [search_result.get_text(separator=\"\\n\") for search_result in lyrics]\n",
    "    lyrics = '\\n'.join(lyrics)\n",
    "    return lyrics\n",
    "all_lyrics = [get_lyrics(url, useragent) for url in links]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "a2288a54-0f5e-42ba-9800-6a728ef6479a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Intro]\n",
      "I'm cryin’ 'cause I love you, oh\n",
      "(Ya ya ya, ya ya, ya ya)\n",
      "[Verse 1]\n",
      "Never been in love before\n",
      "What the fuck are fucking feelings, yo?\n",
      "Once upon a time, I was a ho\n",
      "I don't even wanna ho no mo’\n",
      "Got you something from the liquor store\n",
      "Little bit of Lizzo and some Mo\n",
      "Tryna open up a little mo'\n",
      "Sorry if my heart a little slow\n",
      "[Chorus]\n",
      "I thought that I didn't care\n",
      "I thought I was love-impaired\n",
      "But baby, baby\n",
      "I don't know what I'm gon' do\n",
      "I'm cryin' ’cause I love you, oh\n",
      "Yes, you (Ya ya, ya ya)\n",
      "[Verse 2]\n",
      "Got me standing in the rain\n",
      "Gotta get my hair pressed again\n",
      "I would do it for you all, my friend\n",
      "Ready, baby? Will you be my man?\n",
      "Wanna put you on a plane\n",
      "Fly you out to wherever I am\n",
      "Catch you on the low, I was ashamed\n",
      "Now I’m crazy, 'bout to tat your name\n",
      "[Chorus]\n",
      "I thought that I didn’t care\n",
      "I thought I was love-impaired\n",
      "But baby, oh baby\n",
      "I don't know what I'm gon' do\n",
      "I’m cryin' 'cause I love you, yeah\n",
      "I'm cryin', hey\n",
      "[Chorus]\n",
      "I thought that I didn't care\n",
      "I thought I was love-impaired\n",
      "But baby, baby\n",
      "I don't know what I'm gon' do\n",
      "I'm cryin' 'cause I love you\n",
      "Oh yes, you\n",
      "[Outro]\n",
      "I'm cryin', I'm cryin', I'm cryin', oh\n",
      "I'm cryin' 'cause I love you\n",
      "\n",
      "\n",
      "[Intro]\n",
      "Uh (\n",
      "Go\n",
      ")\n",
      "[Verse 1]\n",
      "Woke up feelin' like I just might run for President\n",
      "Even if there ain't no precedent, switchin' up the messaging\n",
      "I'm about to add a little estrogen\n",
      "Buy my whip by myself, pay my rent by myself\n",
      "Only exes that I care about are in my fucking chromosomes\n",
      "I don't really need you, I'm Macaulay Culkin, home alone\n",
      "Bad bitch, diamonds in my collarbone (Yee, yee)\n",
      "Buy my whip by myself, pay my rent by myself\n",
      "[Pre-Chorus]\n",
      "Sugar, spice, and I'm nice\n",
      "Show me what you're made of\n",
      "Crazy, sexy, cool, baby\n",
      "With or without makeup\n",
      "Got nothing to prove (Got nothing to prove)\n",
      "But I'ma show you how I do\n",
      "[Chorus]\n",
      "Find me up in Magic City bustin' hundreds by the bands\n",
      "And I throw it (Like a girl)\n",
      "Go and throw it (Like a girl)\n",
      "Hangin' out the 750, feelin' bossy in my city\n",
      "'Cause I run it (Like a girl)\n",
      "Run it, run it (Like a girl)\n",
      "I work my femininity\n",
      "I make these boys get on their knees\n",
      "Now watch me do it, watch me do it\n",
      "Look it, look it, I'ma do it\n",
      "Like a girl (Like a girl)\n",
      "Like a girl (Like a girl)\n",
      "[Verse 2]\n",
      "Chaka Kahn, \"Through the Fire,\" light the kerosene (We can do it)\n",
      "Lauryn Hill told me everything is everything (We can do it)\n",
      "Serena Willy showed me I can win the Wimbledon (We can do it)\n",
      "Uh, put me on a pedestal, bet on me, bet I will\n",
      "[Pre-Chorus]\n",
      "Sugar, spice, and I'm nice\n",
      "Show me what you're made of\n",
      "Crazy, sexy, cool, baby\n",
      "With or without makeup\n",
      "Got nothing to prove (Got nothing to prove)\n",
      "But I'ma show you how I do (But I'ma show you how I do)\n",
      "[Chorus]\n",
      "Find me up in Magic City bustin' hundreds by the bands\n",
      "And I throw it (Like a girl)\n",
      "Go and throw it (Like a girl)\n",
      "Hangin' out the 750, feelin' bossy in my city\n",
      "'Cause I run it (Like a girl)\n",
      "Run it, run it (Like a girl)\n",
      "I work my femininity\n",
      "I make these boys get on their knees\n",
      "Now watch me do it, watch me do it\n",
      "Look it, look it, I'mma do it\n",
      "Like a girl (Like a girl)\n",
      "Like a girl (Like a girl)\n",
      "[Bridge]\n",
      "Look, so if you fight like a girl, cry like a girl\n",
      "Do your thing, run the whole damn world\n",
      "If you feel like a girl, then you real like a girl\n",
      "Do your thing, run the whole damn world\n",
      "If you fight like a girl, cry like a girl\n",
      "Do your thing, run the whole damn world\n",
      "If you feel like a girl, then you real like a girl\n",
      "Do your thing, run the whole damn world\n",
      "[Chorus]\n",
      "Find me up in Magic City bustin' hundreds by the bands\n",
      "And I throw it (Like a girl)\n",
      "(Yeah, yeah, yeah, yeah, yeah, yeah)\n",
      "Go and throw it (Like a girl)\n",
      "Hangin' out the 750 (Hangin' out the 750, baby)\n",
      "Feelin' bossy in my city\n",
      "'Cause I run it (Like a girl) (Yeah, I run it like a girl)\n",
      "Run it, run it (Like a girl)\n",
      "I work my femininity (Yeah, I work it baby)\n",
      "I make these boys get on their knees (Get on your knees, yeah)\n",
      "Now watch me do it, watch me do it\n",
      "Look it, look it, I'ma do it (Watch me do it)\n",
      "Like a girl (Like a girl)\n",
      "Like a girl (Like a girl, yeah)\n",
      "[Outro]\n",
      "Shit, fuck, I didn't know it was ending right there\n",
      "\n",
      "\n",
      "[Verse 1]\n",
      "Mirror, mirror on the wall\n",
      "Don't say it ’cause I know I'm cute (Ooh, baby)\n",
      "Louis down to my drawers\n",
      "LV all on my shoes (Ooh, baby)\n",
      "I be drippin' so much sauce\n",
      "Got a bih lookin’ like RAGÚ (Ooh, baby)\n",
      "Lit up like a crystal ball\n",
      "That's cool, baby, so is you\n",
      "That's how I roll\n",
      "[Pre-Chorus]\n",
      "If I'm shinin', everybody gonna shine\n",
      " \n",
      "(Yeah, I'm goals)\n",
      "I was born like this, don't even gotta try\n",
      " \n",
      "(Now you know)\n",
      "I'm like chardonnay, get better over time \n",
      "(So you know)\n",
      "Heard you say I’m not the baddest, bitch, you lie (Haha)\n",
      "[Chorus]\n",
      "It ain’t my fault that I'm out here gettin’ loose\n",
      "Gotta blame it on the Goose\n",
      "Gotta blame it on my juice, baby\n",
      "It ain't my fault that I'm out here makin' news\n",
      "I’m the pudding in the proof\n",
      "Gotta blame it on my juice\n",
      "Ya-ya-ee, ya-ya-ee, ya-ya-ee, ya-ya-ee\n",
      "Blame it on my juice, blame it, blame it on my juice\n",
      "Ya-ya-ee, ya-ya-ee, ya-ya-ee, ya-ya-ee\n",
      "Blame it on my juice, blame it, blame it on my juice (Ooh, baby)\n",
      "[Verse 2]\n",
      "No, I'm not a snack at all\n",
      "Look, baby, I'm the whole damn meal (Ooh, baby)\n",
      "David, you ain't bein' slick\n",
      "Don't dare try to cop a feel (Ooh, baby)\n",
      "The juice ain't worth the squeeze\n",
      "If the juice don't look like this\n",
      " \n",
      "(Like this, like this, like this)\n",
      "Hold up, nigga, please\n",
      "Don't make me have to take your bitch, shit \n",
      "(How I roll)\n",
      "[Pre-Chorus]\n",
      "If I'm shinin', everybody gonna shine\n",
      " \n",
      "(Yeah, I'm goals)\n",
      "I was born like this, don't even gotta try\n",
      " \n",
      "(Now you know)\n",
      "I'm like chardonnay (Okay), get better over time\n",
      " \n",
      "(So you know)\n",
      "Heard you say I'm not the baddest, bitch, you lie \n",
      "(You lie)\n",
      "[Chorus]\n",
      "It ain't my fault that I'm out here gettin' loose\n",
      "Gotta blame it on the Goose\n",
      "Gotta blame it on my juice, baby\n",
      "It ain't my fault that I'm out here makin' news\n",
      "I'm the pudding in the proof\n",
      "Gotta blame it on my juice\n",
      "Ya-ya-ee (Ya-ya-ee), ya-ya-ee, ya-ya-ee, ya-ya-ee\n",
      "Blame it on my juice, blame it, blame it on my juice\n",
      "Ya-ya-ee (Ya-ya-ee), ya-ya-ee, ya-ya-ee, ya-ya-ee\n",
      "Blame it on my juice, blame it, blame it on my juice (Alright)\n",
      "Ya-ya-ee\n",
      "[Bridge]\n",
      "Somebody come get this man\n",
      "I think he got lost in my DMs, what? My DMs, what?\n",
      "You better come get your man\n",
      "I think he wanna be way more than friends, what?\n",
      "More than friends\n",
      "What you want me to say?\n",
      "[Chorus]\n",
      "It ain't my fault that I'm out here gettin' loose\n",
      "Gotta blame it on the Goose\n",
      "Gotta blame it on my juice, baby\n",
      "It ain't my fault that I'm out here makin' news\n",
      "I'm the pudding in the proof (Puddin' in the proof)\n",
      "Gotta blame it on my juice (Blame it on my juice)\n",
      "Ya-ya-ee (Yay-ya), ya-ya-ee, ya-ya-ee, ya-ya-ee\n",
      "Blame it on my juice, blame it, blame it on my juice\n",
      "Ya-ya-ee (Ya-ya-ee), ya-ya-ee, ya-ya-ee, ya-ya-ee\n",
      "Blame it on my juice, blame it, blame it on my juice\n",
      "(Blame it on it, yeah)\n",
      "\n",
      "\n",
      "[Intro]\n",
      "True story\n",
      "No glory, let's go\n",
      "[Verse 1]\n",
      "Yeah, the old me used to love a Gemini\n",
      "Like a threesome, fuckin' with him every night\n",
      "A lotta two-faced people show me both sides\n",
      "So I figured out I gotta be my own type\n",
      "[Pre-Chorus 1]\n",
      "They used to say to get a man, you had to know how to look\n",
      "They used to say to keep a man, you had to know how to cook\n",
      "But I'm solo in Soho, sippin' Soju in Malibu\n",
      "It's a me, myself kinda attitude\n",
      "[Chorus]\n",
      "'Cause I'm my own soulmate (Yeah, yeah)\n",
      "I know how to love me (Love me)\n",
      "I know that I'm always gonna hold me down\n",
      "Yeah, I'm my own soulmate (Yeah, yeah)\n",
      "No, I'm never lonely (Lonely)\n",
      "I know I'm a queen, but I don't need no crown\n",
      "Look up in the mirror like \"Damn, she the one\"\n",
      "[Post-Chorus]\n",
      "One, one, one, one, one, one (The one)\n",
      "Like \"Damn, she the one\"\n",
      "One, one, one (The one)\n",
      "Bad bitch in the mirror like \"Yeah, I'm in love\"\n",
      "Love, love, love, love, love, love (In love)\n",
      "Like \"Yeah, I'm in love\"\n",
      "Love, love, love (In love)\n",
      "Look up in the mirror like \"Damn, she the one\"\n",
      "[Verse 2]\n",
      "And she never tell me to exercise\n",
      "We always get extra fries\n",
      "And you know the sex is fire\n",
      "And I gotta testify\n",
      "I get flowers every Sunday\n",
      "I'ma marry me one day\n",
      "[Pre-Chorus 2]\n",
      "True love ain't something you can buy yourself\n",
      "True love finally happens when you by yourself\n",
      "So if you by yourself, then go and buy yourself\n",
      "Another round from the bottle on the higher shelf\n",
      "[Chorus]\n",
      "'Cause I'm my own soulmate (Yeah, yeah)\n",
      "I know how to love me (Love me)\n",
      "I know that I'm always gonna hold me down\n",
      "(Always gonna hold me down)\n",
      "Yeah, I'm my own soulmate (Yeah, yeah)\n",
      "No, I'm never lonely (Lonely)\n",
      "(No, I'm never lonely, no)\n",
      "I know I'm a queen, but I don't need no crown\n",
      "Look up in the mirror like \"Damn, she the one\"\n",
      "[Post-Chorus]\n",
      "One, one, one, one, one, one (The one)\n",
      "(She the one, she the one)\n",
      "Like, \"Damn, she the one\"\n",
      "(She the one, she the one, she the one)\n",
      "One, one, one (The one)\n",
      "Bad bitch in the mirror like, \"Yeah, I'm in love\"\n",
      "(I'm in love, I'm in love, I'm in love)\n",
      "Love, love, love, love, love, love (In love)\n",
      "(I'm in love, I'm in love)\n",
      "Like \"Yeah, I'm in love\"\n",
      "(I'm in love, yeah, I'm in love)\n",
      "Love, love, love (In love)\n",
      "Look up in the mirror like \"Damn, she the one\"\n",
      "[Bridge]\n",
      "I'm the one (Ooh)\n",
      "I'm the one (You're the one)\n",
      "I'm the one (Oh, you're the one)\n",
      "Look up in the mirror like \"Damn, she the one\"\n",
      "One, one, one, one, one, one\n",
      "Like \"Damn, she the one\"\n",
      "One, one, one, one\n",
      "Bad bitch in the mirror like \"Yeah, I'm in love\"\n",
      "Love, love, love, love, love, love (In love)\n",
      "(I'm in love, yeah, I'm in love)\n",
      "Love, love, love (In love)\n",
      "Look up in the mirror (Ow)\n",
      "[Chorus]\n",
      "'Cause I'm my own soulmate (Yeah, yeah)\n",
      "I know how to love me (Love me)\n",
      "I know that I'm always gonna hold me down\n",
      "Look up in the mirror like \"Damn, she the one\"\n",
      "Yeah, I'm my own soulmate (Yeah, yeah)\n",
      "No, I'm never lonely (Lonely)\n",
      "I know I'm a queen, but I don't need no crown\n",
      "Look up in the mirror like \"Damn, she the one\"\n",
      "\n",
      "\n",
      "[Intro]\n",
      "Woo, oh\n",
      "Here we go again\n",
      "Uh-uh, \n",
      "hee-hee\n",
      "Uh, look, listen, shut up\n",
      "[Chorus]\n",
      "Jerome, Jerome\n",
      "Take your ass home\n",
      "And come back when you're grown\n",
      "Jerome, Jerome\n",
      "Go on, take your ass home\n",
      "Where the peaches have thorns, ooh\n",
      "[Verse 1]\n",
      "Poor little baby\n",
      "Who told you that you stood a chance with this royalty?\n",
      "You're so sweet, bless your heart\n",
      "Can't let a pretty face distract me from business\n",
      "And God as my witness, your ugly ass won't either\n",
      "I'm sorry, 2 AM photos with smileys and hearts\n",
      "Ain't the way to my juicy parts\n",
      "Boy, thank me later, looks good on paper\n",
      "But love isn't easy, so I'll do the hardest part\n",
      "[Chorus]\n",
      "Jerome, Jerome\n",
      "Go on, take your ass home\n",
      "And come back when you're grown\n",
      "Jerome, ooh, Jerome\n",
      "Go on, take your ass home\n",
      "Where the peaches have thorns\n",
      "[Verse 2]\n",
      "I never said I was perfect\n",
      "Or you don't deserve a good person to carry your \n",
      "baggage\n",
      "I know a few girls that can handle it\n",
      "I ain't that kind of chick, but I can call 'em for you if you want\n",
      "I never said that you wasn't attractive\n",
      "Your style and that beard, ooh, don't get me distracted\n",
      "I'm tryna be patient, and patience takes practice\n",
      "The fact is I'm leaving, so just let me have this\n",
      "[Chorus]\n",
      "Jerome, Jerome\n",
      "Take your ass home\n",
      "And come back when you're grown, oh\n",
      "Jerome, ooh, Jerome\n",
      "Go on, take your ass home\n",
      "Where the peaches have thorns\n",
      "[Outro]\n",
      "Hee, hee, hee, hee, hee\n",
      "(Don't cry for me, baby)\n",
      "Hee, hee, hee, hee, hee\n",
      "Hee, hee, hee, hee, hee\n",
      "(Oh, don't cry for me, baby)\n",
      "Hee, hee, hee, hee, hee, hee\n",
      "\n",
      "\n",
      "[Verse 1]\n",
      "Pull this car over, babe\n",
      "Don't pretend like you don't know\n",
      "A lot of girls have time for this shit\n",
      "Honestly, I don't\n",
      "[Pre-Chorus 1]\n",
      "He said, \"Why you cryin', baby?\n",
      "Why you cryin', baby?\n",
      "What did I do?\" Oh\n",
      "I said, shut up and kiss me, oh\n",
      "'Cause I don't wanna be here for long\n",
      "[Chorus]\n",
      "I swore you'd never see this side\n",
      "But it's so hard to say goodbye\n",
      "I don't need to apologize\n",
      "Us big girls gotta cry\n",
      "[Post-Chorus]\n",
      "Ooh, ooh, ooh\n",
      "Ooh, ooh, ooh\n",
      "[Verse 2]\n",
      "You should feel honored boy, uh\n",
      "You got me feelin' this much\n",
      "I clear my schedule for you\n",
      "Let my guard down for you\n",
      "And you gon' make me put it back up\n",
      "[Pre-Chorus 2]\n",
      "Oh, you know why I'm cryin'\n",
      "Why I'm cryin', baby\n",
      "You don't pick up the phone\n",
      "You do the same thing to your friends and family\n",
      "You gon' end up alone\n",
      "[Chorus]\n",
      "I swore you'd never see this side\n",
      "But it's so hard to say goodbye\n",
      "I don't need to apologize\n",
      "Us big girls got to cry\n",
      "[Post-Chorus]\n",
      "Ooh, ooh, ooh\n",
      "Ooh, ooh, ooh\n",
      "[Bridge]\n",
      "It feels so good, so good sometimes\n",
      "Some big girls gotta cry\n",
      "After this is done, just take me home\n",
      "I'm feelin' vulnerable\n",
      "I don't need to apologize\n",
      "Us big girls gotta cry\n",
      "[Outro]\n",
      "Oh, oh, oh\n",
      "Oh, oh, oh\n",
      "Oh, oh, oh\n",
      "Oh, oh, oh\n",
      "Ooh, it's my life\n",
      "You know it hurts sometimes\n",
      "My love is your love\n",
      "Why can't your love be mine?\n",
      "Ooh, ooh\n",
      "Oh, I get a little lost sometimes\n",
      "Sometimes you need to cry\n",
      "\n",
      "\n",
      "[Intro: Lizzo]\n",
      "I've been waitin' for this one\n",
      "Turn it up\n",
      "[Chorus: Lizzo]\n",
      "Slow songs, they for skinny hoes\n",
      "Can't move all of this here to one of those\n",
      "I'm a thick bitch, I need tempo (\n",
      "Tempo\n",
      ")\n",
      "Fuck it up to the tempo\n",
      "[Verse 1: Lizzo]\n",
      "Pitty-pat, pitty-pat, pitty-pitty-pat (\n",
      "Pat\n",
      ")\n",
      "Look at my ass, it's fitty-fitty-fat (\n",
      "Fat\n",
      ")\n",
      "Kitty cat, kitty cat, kitty-kitty cat (\n",
      "Cat\n",
      ")\n",
      "Prrr me a glass, boy, I like my water wet (\n",
      "Wet\n",
      ")\n",
      "Throw it back (\n",
      "Throw it back, threw it back\n",
      ")\n",
      "Catch that (\n",
      "Catch that, catch that\n",
      ")\n",
      "I need a jack (\n",
      "Woo\n",
      ")\n",
      "For all of this ass, but it won't go flat\n",
      " (\n",
      "Wait a minute\n",
      ")\n",
      "Baby, baby (\n",
      "Baby\n",
      ")\n",
      "Come eat some of this cakey (\n",
      "Cake\n",
      ")\n",
      "He look like he could gain a little weight (\n",
      "A little weight\n",
      ")\n",
      "Lick the icing off, put the rest in your face (\n",
      "Face\n",
      ")\n",
      "[Chorus: Lizzo]\n",
      "Slow songs, they for skinny hoes\n",
      "Can't move all of this here to one of those (\n",
      "Hey\n",
      ")\n",
      "I'm a thick bitch, I need tempo (\n",
      "Need it\n",
      ")\n",
      "Fuck it up to the tempo\n",
      "Fuck it up to the tempo\n",
      "Fuck it up to the tempo\n",
      "Slow songs, they for skinny hoes\n",
      "Fuck it up to the tempo (\n",
      "Tempo\n",
      ")\n",
      "[Verse 2: Lizzo]\n",
      "Fuck it up, fuck it up\n",
      "Boyfriend watchin', oh now he wanna knuckle up\n",
      "Get on this ride, baby, you gon' have to buckle up\n",
      "Thick thighs save lives\n",
      ", \n",
      "call me little buttercup\n",
      "All means necessary (Ayy)\n",
      "My ass is not an accessorary (\n",
      "What?\n",
      ")\n",
      "Yeah, I said it, accessorary (\n",
      "What?\n",
      ")\n",
      "Twerk skills up on legendary\n",
      "[Chorus: Lizzo]\n",
      "Slow songs, they for skinny hoes\n",
      "Can't move all of this here to one of those (\n",
      "Hey\n",
      ")\n",
      "I'm a thick bitch, I need tempo (\n",
      "Need it\n",
      ")\n",
      "Fuck it up to the tempo\n",
      "Fuck it up to the tempo\n",
      "Fuck it up to the tempo\n",
      "Slow songs, they for skinny hoes\n",
      "Fuck it up to the tempo (Missy)\n",
      "[Verse 3: Missy Elliott]\n",
      "Let's go (Let's go), let's go (Let's go)\n",
      "All the fly girls head to the dance floor\n",
      "Kitty cat, kitty cat, prrr\n",
      "All the thick girls down on the flrrr (On the floor)\n",
      "Ice on my neck like brrr (Like brrr)\n",
      "I'm big-boned with nice curves (Nice curves)\n",
      "Look at me, I know I look grrrd (Good)\n",
      "Look good, look good, look grrrd (Look good)\n",
      "I'ma show y'all chicks how to drrr it\n",
      "Your nigga, he wish he could\n",
      "Thick girl (Thick girl), split (Split)\n",
      "Get a bag (Get a bag), get rich (Get rich)\n",
      "Hair done (Hair done), fixed (Fixed)\n",
      "If you see a hater, tell him quit (Stop)\n",
      "Get your own dough (Own dough)\n",
      "Get your bread, own dough (Own dough)\n",
      "Go on, ladies, head to the floor (Floor)\n",
      "Fuck up the tempo, thick girls get low (Woo)\n",
      "[Chorus: Lizzo]\n",
      "Slow songs, they for skinny hoes\n",
      "Can't move all of this here to one of those (\n",
      "Hey\n",
      ")\n",
      "I'm a thick bitch, I need tempo (\n",
      "Need it\n",
      ")\n",
      "Fuck it up to the tempo\n",
      "Fuck it up to the tempo\n",
      "Fuck it up to the tempo\n",
      "Slow songs, they for skinny hoes\n",
      "Fuck it up to the tempo\n",
      "[Outro: Lizzo]\n",
      "Girl, run that shit back\n",
      "\n",
      "\n",
      "[Chorus: Lizzo]\n",
      "That's exactly how I feel\n",
      "That's exactly how I feel\n",
      "That's exactly how I feel\n",
      "[Verse 1: Lizzo]\n",
      "I woke up this morning (God)\n",
      "Wrong side of the bed\n",
      "I don't have to 'splain it, oh (How I feel)\n",
      "I might be a bitch (Bitch)\n",
      "I might make a friend (Friend)\n",
      "Ain't I so amazing? (Uh-huh)\n",
      "[Pre-Chorus: Lizzo]\n",
      "Love me or hate me\n",
      "Ooh, I ain't changing\n",
      "And I don't give a fuck, no\n",
      "[Chorus: Lizzo]\n",
      "That's exactly how I feel (Uh)\n",
      "That's exactly how I feel (Say it again)\n",
      "That's exactly how I feel (Woo)\n",
      "[Verse 2: Lizzo & \n",
      "Gucci Mane\n",
      "]\n",
      "Don't hide no emotions (Emotions)\n",
      "Wear 'em on my sleeve (On my sleeve)\n",
      "All my feelings Gucci (\n",
      "It's Gucci\n",
      ", haha)\n",
      "Can't hold back my tears\n",
      "That would be a crime\n",
      "'Cause I look pretty cryin'\n",
      "Oh, they ain't tell you?\n",
      "[Pre-Chorus: Lizzo]\n",
      "Love me or hate me\n",
      "Ooh, I ain't changing\n",
      "And I don't give a fuck, no\n",
      "[Chorus: Lizzo & \n",
      "Gucci Mane\n",
      "]\n",
      "That's exactly how I feel (How I feel, \n",
      "yeah\n",
      ")\n",
      "That's exactly how I feel (How I feel, \n",
      "it's Gucci\n",
      ")\n",
      "That's exactly how I feel\n",
      "[Verse 3: Gucci Mane]\n",
      "I feel like hurtin' they feelings (Yeah)\n",
      "I don't get mad, I get millions (Yeah)\n",
      "My heart ain't got no feelings\n",
      "My cars ain't got no ceiling\n",
      "I ain't with that acting all friendly\n",
      "Please step away from the Bentley\n",
      "You're either not with me or with me\n",
      "The fye close by like it's Whitney\n",
      "Stay in your lane, don't tempt me\n",
      "Mr. Bust-A-Whole-Clip-'Til-It's-Empty\n",
      "I got a feelin' that tonight's gonna be a good night\n",
      "Like Ferg and them said, god willing\n",
      "Fire chains on like burr (Burr)\n",
      "Feelin' like rockin' my fur (Fur)\n",
      "Rolls Royce with the chauffeur (Huh?)\n",
      "How your toes feel on a mink fur?\n",
      "It's Gucci\n",
      "[Pre-Chorus: Lizzo]\n",
      "Cry 'cause I want to\n",
      "Smile if I want to, huh\n",
      "Yes, I can\n",
      "Love 'cause I want to\n",
      "Get so mad I could scream (I could scream)\n",
      "I could scream (I could scream)\n",
      "I could scream (Sing it)\n",
      "Yeah, yeah, yeah, yeah\n",
      "[Chorus: Lizzo]\n",
      "That's exactly how I feel (How I feel)\n",
      "That's exactly how I feel (How I feel)\n",
      "That's exactly how I feel\n",
      "\n",
      "\n",
      "[Verse 1]\n",
      "Black, white, ebony all sound good to me\n",
      "Two tone recipe, got good chemistry\n",
      "J. F. Kennedys, kiss hood celebrities\n",
      "Don't matter to me 'cause I like everything\n",
      "[Chorus]\n",
      "You can be my lover\n",
      "'Cause love looks better in color\n",
      "(Yeah, yeah, yeah, yeah, yeah, \n",
      "tell 'em\n",
      ")\n",
      "[Verse 2]\n",
      "Big dick energy (Tell 'em)\n",
      "Tastes like collard greens (Tell 'em)\n",
      "Big ass patty cake (Tell 'em)\n",
      "Hey, happy days (Tell 'em)\n",
      "Bitch, don't label me (Tell 'em)\n",
      "I'm me, famously (Tell 'em)\n",
      "Nobody's safe, nobody\n",
      "You and you and you and\n",
      "[Chorus]\n",
      "You can be my lover\n",
      "'Cause love looks better in color\n",
      "(Yeah, yeah, yeah, yeah, yeah)\n",
      "[Post-Chorus]\n",
      "Color\n",
      "Love looks better\n",
      "Love, love looks better (You know it do)\n",
      "In color, ooh\n",
      "If I try to tell you, then you try to hurt me\n",
      "Ya'll don't hear me\n",
      "[Verse 3]\n",
      "Sick six piston (Tell 'em)\n",
      "Tiptoe kissin' (Tell 'em)\n",
      "Catcall kittens (Tell 'em)\n",
      "Whole milk lickin' (Tell 'em)\n",
      "See my vision (Tell 'em)\n",
      "Rainbow smitten (Tell 'em)\n",
      "It's easy, baby, got the whole world for the pickin'\n",
      "[Chorus]\n",
      "You can be my lover\n",
      "'Cause love looks better in color\n",
      "(Yeah, yeah, yeah, yeah, yeah)\n",
      "[Post-Chorus]\n",
      "Color\n",
      "My whole world o' color, babe\n",
      "Love looks better\n",
      "Love, love looks better\n",
      "In color, ooh (Rainbows and shit)\n",
      "Oh, if I try to tell you then you try to hurt me\n",
      "I'ma tell your momma\n",
      "[Bridge]\n",
      "Color me, color me, color me, color me\n",
      "Color me, color me, color me, color me\n",
      "Color me, color me, color me, color me\n",
      "Color me, color me, color me, color me\n",
      "[Chorus]\n",
      "Yeah, you can be my lover\n",
      "'Cause love looks better in color\n",
      "(Yeah, yeah, yeah, yeah, yeah)\n",
      "It's dark under the covers\n",
      "But love looks better in color\n",
      "\n",
      "\n",
      "[Chorus]\n",
      "Heaven, help me\n",
      "If love ain't dead, I'ma kill it 'cause it's killin' me\n",
      "Cold world, they'll be livin' in a fantasy\n",
      "Got me, it's the only thing I'll ever need\n",
      "Heaven, help me\n",
      "If love ain't dead, I'ma kill it 'cause it's killin' me\n",
      "Love is a drug, methamphetamine\n",
      "D-A-R-E, they'll be hooked on me\n",
      "[Verse 1]\n",
      "Y'all, who you think you sassing? (Sassing)\n",
      "Say whoa, baby, I'm a classic (Classic)\n",
      "Say whoa, something like a Hitchcock\n",
      "Say whoa, got you goin' psycho (Psycho)\n",
      "Oh no, don't you get your shit rocked\n",
      "Say whoa, \n",
      "Kesha with the tick tock\n",
      "Say whoa, \n",
      "time's up on these motherfuckers\n",
      "Say whoa, what you say, what you say, sucker?\n",
      "(What you say? What you say?)\n",
      "Say whoa\n",
      "[Pre-Chorus]\n",
      "If you think you got me dickmatized\n",
      "(Oh yeah, oh yeah, oh yeah, oh yeah)\n",
      "I need to get you out of my life\n",
      "Can I get an amen?\n",
      "[Chorus]\n",
      "Heaven, help me\n",
      "If love ain't dead, I'ma kill it 'cause it's killin' me\n",
      "Cold world, they'll be livin' in a fantasy\n",
      "Got me, it's the only thing I'll ever need\n",
      "Heaven, help me\n",
      "If love ain't dead, I'ma kill it 'cause it's killin' me\n",
      "Love is a drug, methamphetamine\n",
      "D-A-R-E, they'll be hooked on me\n",
      "[Verse 2]\n",
      "Hit 'em with the bye bye (Bye bye), say whoa\n",
      "Hit 'em with the drive-by (Brrr), say whoa\n",
      "Hit 'em with the dry cry (Dry cry), say whoa\n",
      "Then you get a new guy, hello\n",
      "[Pre-Chorus]\n",
      "Even if you are the love of my life\n",
      "(Oh yeah, oh yeah, oh yeah, oh yeah)\n",
      "Sorry, \n",
      "baby, I got too much pride\n",
      "[Chorus]\n",
      "Heaven, help me\n",
      "If love ain't dead, I'ma kill it 'cause it's killin' me\n",
      "Cold world, they'll be livin' in a fantasy\n",
      "Got me, it's the only thing I'll ever need\n",
      "Heaven, help me\n",
      "If love ain't dead, I'ma kill it 'cause it's killin' me\n",
      "Love is a drug, methamphetamine\n",
      "D-A-R-E, they'll be hooked on me\n",
      "[Post-Chorus]\n",
      "Heaven, help me\n",
      "If love ain't dead, I'ma kill it 'cause it's killin' me\n",
      "Ooh, can I get an amen? (Amen)\n",
      "[Outro]\n",
      "Been thinkin' about you, thinkin' about you\n",
      "Thinkin' about you baby\n",
      "If love didn't drive me crazy, maybe I'd be your baby\n",
      "Trust, baby, you can't make me beg or try to change, hey\n",
      "\n",
      "\n",
      "[Intro]\n",
      "(Yeah)\n",
      "Damn\n",
      "[Verse 1]\n",
      "Hair down, moonlit\n",
      "Look at my lipstick\n",
      "So thick, so fit\n",
      "Wanna put your lips in places, ooh\n",
      "The sun don't shine there\n",
      "I like that right there (I like that)\n",
      "Keep goin’ and goin' and keep it flowin', uh\n",
      "You know what you doin’, don't you?\n",
      "[Chorus]\n",
      "So I lounge around in my lingerie\n",
      "I wanna be prepared for you just in case\n",
      "So I lounge around in my lingerie\n",
      "You better come my way\n",
      "[Verse 2]\n",
      "I don't got no secrets you don't know\n",
      "These panties are see-through, I'm exposed, yeah\n",
      "Treat you with my body, my eyes closed\n",
      "Baby, let me feel you close\n",
      "You make me crescendo, I'm going up\n",
      "Ooh, ooh, ooh, yeah\n",
      "Let's open the window\n",
      "And let the world know\n",
      "Oh, oh, oh\n",
      "[Chorus]\n",
      "So I lounge around in my lingerie\n",
      "I wanna be prepared for you just in case\n",
      "So I lounge around in my lingerie\n",
      "You better come my way\n",
      "[Bridge]\n",
      "Yeah, yeah, ow\n",
      "Ooh, ooh-ooh, ooh\n",
      "Ooh, ooh-ooh, ooh\n",
      "Ooh, ooh-ooh, ooh\n",
      "Ooh, ooh-ooh, ooh\n",
      "[Chorus]\n",
      "So I lounge around in my lingerie\n",
      "I wanna be prepared for you just in case, yeah\n",
      "So I lounge around in my lingerie\n",
      "Ooh, you better come my way\n",
      "\n",
      "\n",
      "[Verse 1]\n",
      "Ayy, boy, whatcha say, boy?\n",
      "You tryna play coy like a Gameboy?\n",
      "Hit my phone boy, is you home boy?\n",
      "Are you alone boy? Come give me dome, boy\n",
      "Got a boy with degrees, a boy in the streets\n",
      "A boy on his knees, he a man in the sheets\n",
      "Sheesh, it's all Greek to me\n",
      "Got this boy speaking Spanish, ay, papi\n",
      "[Pre-Chorus]\n",
      "Baby, I don't need you\n",
      "I just wanna freak you\n",
      "I heard you a freak too\n",
      "What's two plus two? (Four, three, two, ow)\n",
      "[Chorus]\n",
      "Boys, boys, boys\n",
      "Make a girl go crazy (\n",
      "Four, three, two, ow\n",
      ")\n",
      "Boys, boys, boys\n",
      "Make a girl go crazy (\n",
      "Four, three\n",
      ")\n",
      "[Verse 2]\n",
      "I like big boys, itty bitty boys\n",
      "Mississippi boys, inner city boys\n",
      "I like the pretty boys with the bow tie\n",
      "Get your nails did, let it blow dry\n",
      "I like a big beard, I like a clean face\n",
      "I don't discriminate, come and get a taste\n",
      "From the playboys to the gay boys\n",
      "Go and slay, boys, you my fave boys\n",
      "[Pre-Chorus]\n",
      "Baby, I don't need you (I don't need)\n",
      "I just wanna freak you (Want, I want it bad)\n",
      "I heard you a freak too (That's right)\n",
      "What's two plus two? (\n",
      "Four, three, two, ow\n",
      ")\n",
      "[Chorus]\n",
      "Boys, boys, boys\n",
      "Make a girl go crazy (\n",
      "Four, three, two, ow\n",
      ")\n",
      "Boys, boys, boys\n",
      "Make a girl go crazy (\n",
      "Four, three, two, ow\n",
      ")\n",
      "[Instrumental]\n",
      "[Chorus]\n",
      "Four, three, two, ow\n",
      " (Woo)\n",
      "Boys, boys, boys\n",
      "Make a girl go crazy (\n",
      "Four, three, two, ow\n",
      ", done lost your mind)\n",
      "Boys, boys, boys\n",
      "Make a girl go crazy, ow\n",
      "\n",
      "\n",
      "[Intro]\n",
      "Why men great 'til they gotta be great?\n",
      "Woo!\n",
      "[Verse 1]\n",
      "I just took a DNA test, turns out I'm 100% that bitch\n",
      "Even when I'm crying crazy\n",
      "Yeah, I got boy problems, that's the human in me\n",
      "Bling bling, then I solve 'em, that's the goddess in me\n",
      "You coulda had a bad bitch, non-committal\n",
      "Help you with your career just a little\n",
      "You're 'posed to hold me down, but you're holding me back\n",
      "And that's the sound of me not calling you back\n",
      "[Chorus]\n",
      "Why men great 'til they gotta be great? ('Til they gotta be great)\n",
      "Don't text me, tell it straight to my face (Tell it straight to my face)\n",
      "Best friend sat me down in the salon chair (Down in the salon chair)\n",
      "Shampoo press, get you out of my hair\n",
      "Fresh photos with the bomb lighting (With the bomb lighting)\n",
      "New man on the Minnesota Vikings (Minnesota Vikings)\n",
      "Truth hurts, needed something more exciting (Yee)\n",
      "Bom bom bi dom bi dum bum bay (Eh, yeah, yeah, yeah)\n",
      "[Verse 2]\n",
      "You tried to break my heart?\n",
      "Oh, that breaks my heart\n",
      "That you thought you ever had it\n",
      "No, you ain't from the start\n",
      "Hey, I'm glad you're back with your bitch\n",
      "I mean, who would wanna hide this?\n",
      "I will never, ever, ever, ever, ever be your side chick\n",
      "I put the sing in single\n",
      "Ain't worried 'bout a ring on my finger\n",
      "So you can tell your friend, \"Shoot your shot\" when you see him\n",
      "It's okay, he already in my DMs\n",
      "[Chorus]\n",
      "Why men great 'til they gotta be great? ('Til they gotta be great)\n",
      "Don't text me, tell it straight to my face (Tell it straight to my face)\n",
      "Best friend sat me down in the salon chair (Down in the salon chair)\n",
      "Shampoo press, get you out of my hair\n",
      "Fresh photos with the bomb lighting (With the bomb lighting)\n",
      "New man on the Minnesota Vikings (Minnesota Vikings)\n",
      "Truth hurts, needed something more exciting (Yee)\n",
      "Bom bom bi dom bi dum bum bay (Eh, yeah, yeah, yeah)\n",
      "[Bridge]\n",
      "I'ma hit you back in a minute (Yeah, yeah)\n",
      "I don't play tag, bitch, I been it (One time)\n",
      "We don't fuck with lies (Two times), we don't do goodbyes (Woo)\n",
      "We just keep it pushing like ay-ay-ay\n",
      "I'ma hit you back in a minute (Yeah, yeah)\n",
      "I don't play tag, bitch, I been it (One time)\n",
      "We don't fuck with lies (Two times), we don't do goodbyes (Woo)\n",
      "We just keep it pushing like ay-ay-ay (Woo)\n",
      "[Chorus]\n",
      "Why men great 'til they gotta be great? ('Til they gotta be great)\n",
      "Don't text me, tell it straight to my face (Tell it straight to my face)\n",
      "Best friend sat me down in the salon chair (Down in the salon chair)\n",
      "Shampoo press, get you out of my hair\n",
      "Fresh photos with the bomb lighting (With the bomb lighting)\n",
      "New man on the Minnesota Vikings (Minnesota Vikings)\n",
      "Truth hurts, needed something more exciting (Yee)\n",
      "Bom bom bi dom bi dum bum bay (Eh, yeah, yeah, yeah)\n",
      "[Outro]\n",
      "With the bomb lighting\n",
      "Minnesota Vikings\n",
      "Yee, eh, yeah, yeah, yeah\n",
      "\n",
      "\n",
      "[Chorus]\n",
      "I am free, yeah yeah\n",
      "Come water me, oh oh\n",
      "Love you so, but if you don't\n",
      "I have to leave, oh no\n",
      "I am free, yeah yeah\n",
      "Come water me, oh oh\n",
      "Love you so, but if you don't\n",
      "I have to leave, oh no\n",
      "[Verse 1]\n",
      "It's hot, I think I wanna kick off both of my socks\n",
      "I can feel it boilin' up in this pot\n",
      "A closed mouth don't get fed, is you hungry or not?\n",
      "Like brr brr, operator\n",
      "Caboose like, choo choo, bless my trainer\n",
      "Thank God, thank God, thank God, I'm gettin' thicker\n",
      "And I got 'em sweatin' in the middle of the winter\n",
      "[Pre-Chorus]\n",
      "(Get up) I don't get dehydrated, I moisturize it daily\n",
      "I am my inspiration, I am my inspiration\n",
      "(Get up) But I love you, no limit, I need to know you in it\n",
      "If you got time, let's spend it\n",
      "[Chorus]\n",
      "I am free, yeah yeah\n",
      "Come water me, oh oh\n",
      "Love you so, but if you don't\n",
      "I have to leave, oh no\n",
      "I am free, yeah yeah\n",
      "Come water me, oh oh\n",
      "Love you so, but if you don't\n",
      "I have to leave, oh no\n",
      "[Verse 2]\n",
      "I think I need a washrag, perspiratin'\n",
      "Gimme that sweet tea, you think I'm playin'\n",
      "I need it all night, no sleep\n",
      "You can get it in the kitchen if you sweatin' out the sheets\n",
      "Like brr brr, operator\n",
      "You got me like, hee hee, Mike Jackson\n",
      "Thank God, thank God, thank God, I'm gettin' thicker\n",
      "And I can make you boil up, baby, let it simmer\n",
      "[Pre-Chorus]\n",
      "(Get up) I don't get dehydrated, I moisturize it daily\n",
      "I am my inspiration, I am my inspiration\n",
      "(Get up) But I love you, no limit, I need to know you in it\n",
      "If you got time, let's spend it\n",
      "[Chorus]\n",
      "I am free, yeah yeah\n",
      "Come water me, oh oh\n",
      "Love you so, but if you don't\n",
      "I have to leave, oh no\n",
      "I am free, yeah yeah\n",
      "Come water me, oh oh\n",
      "Love you so, but if you don't\n",
      "I have to leave, oh no\n",
      "[Bridge]\n",
      "I'm so pretty and so ditty and these boys be after kitty\n",
      "I just snap and pivot, uh huh, uh huh\n",
      "I'm so pretty and so ditty and these boys be after kitty\n",
      "I just snap and pivot, uh huh, uh huh\n",
      "[Pre-Chorus]\n",
      "(Get up) I don't get dehydrated, I moisturize it daily\n",
      "I am my inspiration, I am my inspiration\n",
      "(Get up) But I love you, no limit, I need to know you in it\n",
      "If you got time, let's spend it\n",
      "[Chorus]\n",
      "I am free, yeah yeah\n",
      "Come water me, oh oh\n",
      "Love you so, but if you don't\n",
      "I have to leave, oh no\n",
      "I am free, yeah yeah\n",
      "Come water me, oh oh\n",
      "Love you so, but if you don't\n",
      "I have to leave, oh no\n",
      "\n",
      "\n",
      "[Chorus: Lizzo]\n",
      "I do my hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "Hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "[Verse 1: Lizzo]\n",
      "Woo, child, tired of the bullshit\n",
      "Go on, dust your shoulders off, keep it moving\n",
      "Yes, Lord, tryna get some new shit\n",
      "In there, swimwear, going-to-the-pool shit\n",
      "Come now, come dry your eyes\n",
      "You know you a star, you can touch the sky\n",
      "I know that it's hard but you have to try\n",
      "If you need advice, let me simplify\n",
      "[Pre-Chorus: Lizzo]\n",
      "If he don't love you anymore\n",
      "Just walk your fine ass out the door\n",
      "[Chorus: Lizzo]\n",
      "I do my hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "Hair toss, check my nails\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "(Feeling good as hell)\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "[Verse 2: Lizzo & \n",
      "Ariana Grande\n",
      "]\n",
      "Woo, girl, need to kick off your shoes\n",
      "Gotta take a deep breath, time to focus on you\n",
      "All the big fights, long nights that you been through\n",
      "I got a bottle of tequila I been saving for you (\n",
      "Thank god\n",
      ")\n",
      "'Cause he better know my worth\n",
      "There's so much that I deserve\n",
      "But I ain't worried now, I'ma let my hair down\n",
      "He been tryin' it, but not today\n",
      "[Pre-Chorus: Ariana Grande, \n",
      "Lizzo\n",
      " & \n",
      "Both\n",
      "]\n",
      "So girl, if he don't love you anymore\n",
      "Then walk your fine ass\n",
      " \n",
      "out the door\n",
      "[Chorus: Lizzo, \n",
      "Ariana Grande\n",
      " & \n",
      "Both\n",
      "]\n",
      "And do your hair\n",
      " toss, check my nails (\n",
      "Ah\n",
      ")\n",
      "Baby, how you feelin'? (Feelin' good as hell)\n",
      "Hair toss, check my nails (\n",
      "Mmm\n",
      ")\n",
      "Baby, how you feelin'? (Feelin' good as hell) (Show it off tonight)\n",
      "Hair toss, check my nails (\n",
      "Woo\n",
      ")\n",
      "Baby, how you feelin'? (Feelin' good as hell) (And we gon' be alright, \n",
      "oh\n",
      ")\n",
      "Hair toss, check my nails (\n",
      "Mmm\n",
      ")\n",
      "Baby, how you feelin'? (Feelin' good as hell) (\n",
      "Mmm\n",
      ")\n",
      "[Pre-Chorus: Lizzo & \n",
      "Ariana Grande\n",
      "]\n",
      "Ooh, yeah (\n",
      "Mmm\n",
      "), alright (\n",
      "Yee\n",
      ")\n",
      "Listen, \n",
      "so girl, if he don't love you anymore, more\n",
      "Then walk your fine ass out the door (\n",
      "Door\n",
      ")\n",
      "[Chorus: Lizzo, \n",
      "Ariana Grande\n",
      " & \n",
      "Both\n",
      "]\n",
      "And do your hair toss, check my nails (\n",
      "Woo\n",
      ")\n",
      "Baby, how you feelin'? (Feelin' good as hell) (Woo-hoo)\n",
      "Hair toss, check my nails (\n",
      "Hey yeah\n",
      ")\n",
      "Baby, how you feelin'? (Feelin' good as hell) (Show it off tonight)\n",
      "Hair toss, check my nails (\n",
      "Mmm\n",
      ")\n",
      "Baby, how you feelin'? (Feelin' good as hell) (And we gon' make it right)\n",
      "Hair toss, check my nails (\n",
      "Oh, hey\n",
      ")\n",
      "Baby, how you feelin'? (Feelin' good as hell) (\n",
      "Yeah\n",
      ")\n",
      "(Feelin' good as hell)\n",
      "Baby, how you feelin'? (\n",
      "Feelin' good as hell\n",
      ")\n",
      "(\n",
      "Haha\n",
      ", \n",
      "I danced, that's what I did\n",
      ")\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in all_lyrics:\n",
    "    print(i)\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd37c304-78f7-46a3-993d-1d5e32c68e56",
   "metadata": {},
   "source": [
    "(6) Finding the hidden API:\n",
    "\n",
    "APIs are the primary mechanism for transfering data over the internet, but most APIs are for internal use for a website and are not intended for outside users. When this is the case, there won't be an obvious link to the API and there won't be any documentation. You can still sometimes get access to the API. This exercise will guide you through one instance of finding and using a hidden API.\n",
    "\n",
    "It's the time of year for Spirit Halloween stores to pop up. They often take over the stores where large chains have recently met their demise. Go to the store locator page: https://stores.spirithalloween.com/ Notice that the top hit says \"Former Sears\". I want to know how often Sears appears in the descriptions of these stores, what other dead chains show up, and how frequently. If you right click on this page and view source, the data that appears about the nearby stores does not appear. Instead this website is calling a hidden API that we can access.\n",
    "\n",
    "For this problem, use the webpage inspector in the Mozilla Firefox web browser. If you don't have Firefox, download it here: https://www.mozilla.org/en-US/firefox/new/\n",
    "\n",
    "**Step 1**: With Firefox, go to https://stores.spirithalloween.com/ Right click on this page, and select \"Inspect\". (You can still use Chrome or anything you want for Jupyter Lab, just have Firefox open for the inspect tool).\n",
    "\n",
    "**Step 2**: Inspect is a complicated but extremely useful tool. It reports all APIs and other web-based connections that a website makes. Click on Network to see these connections. These are all the calls the Spirit Halloween website makes to various APIs to display images, maps, ads, and addresses. \n",
    "\n",
    "**Step 3**: In the right-hand window within the Inspect tool, click on Response. Our task is to sift through the various API calls and to look at the responses until we see the specific address information we need. This took me a long, long time, and I'll save you that trouble -- find the Domain maps.spirithalloween.com, click it, and look at the JSON that appears under Response. Scroll down until you see the addresses and the phrase \"Former Sears\". If you do not see maps.spirithalloween.com under Domains, reload the page and look again.\n",
    "\n",
    "**Step 4**: Now that we know the API call that Spirit Halloween used to get the addresses, we can deduce the root, endpoint, and some of the parameters. Hover your mouse over file. It displays: https://maps.spirithalloween.com/api/getAsyncLocations?template=domain&level=domain. So the root should be https://maps.spirithalloween.com, the endpoint should be /api/getAsyncLocations, and two of the parameters should be template and level, both set equal to \"domain\".\n",
    "\n",
    "**Step 5**: Take a closer look at Response. There are other parameters here we can use. At the bottom of the JSON output is a key named \"options\". I didn't know this for sure, but my bet was that the key-value pairs inside \"options\" can be changed as parameters in the call to the API. My goal is to get all of the Spirit Halloween stores, not just the ones near me. So some of the values I wanted to change are lat and lng, which define my location, radius which defines the distance in miles from my location, and limit, which I bet specifies the maximum number of results. I want these values to be 'lat': 40.380028, 'lng': -97.910156, which places my location in the middle of the country, 'radius': 1800 which captures the entire lower 48 states, and 'limit': 1500 which exceeds the number of Spirit Halloween stores (I had to guess-and-check that).\n",
    "\n",
    "For this problem, take it from here. Issue an API call to https://maps.spirithalloween.com with endpoint /api/getAsyncLocations. Provide your user-agent and email address in the headers, and set the parameters as discussed in step 5. Find a way to extract the descriptive phrases such as 'Former Sears' and 'Next to Ashley Furniture' from each store's address and store them in a list. Filter this list to only the elements that contain the word \"Former\". Then report the frequencies. If you saved these phrases as a list named `formerlocs` you can get frequencies by typing:\n",
    "```\n",
    "formerlocsDF = pd.DataFrame()\n",
    "formerlocsDF['formerlocs'] = formerlocs\n",
    "formerlocsDF['formerlocs'].value_counts()\n",
    "```\n",
    "Hint: even though the output is JSON, some of the data inside the JSON is encoded as HTML, so you will need to use `BeautifulSoup` as well as the `json` package. [4 points]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "2f4c01b9-956f-43a3-8023-217c99c2878f",
   "metadata": {},
   "outputs": [],
   "source": [
    "root = 'https://maps.spirithalloween.com'\n",
    "start_date = '2022-10-6'\n",
    "spirit_endpoint = '/api/getAsyncLocations?template=domain&level=domain&\\\n",
    "                    lat=40.380028&lng=-97.910156&radius=1800&limit=1500'\n",
    "r = requests.get(root + spirit_endpoint,\n",
    "                headers = headers)\n",
    "my_html = BeautifulSoup(r.text, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "1c6524f2-af41-4576-ac76-494378f4ff95",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_json = json.loads(r.text)\n",
    "#my_json['markers'][6]['info']\n",
    "formerlocs = []\n",
    "for i in my_json['markers']:\n",
    "    my_html = BeautifulSoup(i['info'], 'html.parser')\n",
    "    formerlocs.append(my_html.find_all(class_='address-two')[0].text)\n",
    "formerlocs = [loc for loc in formerlocs if 'ormer' in loc]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "6dc64c18-be6a-4ec9-8efc-0e74886613cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Former Sears                                      63\n",
       "Former Pier 1                                     32\n",
       "Former Dress Barn                                 25\n",
       "Former Office Depot                               23\n",
       "Former Party City                                 20\n",
       "                                                  ..\n",
       "Former Jo Anns Fabrics                             1\n",
       "Former North Face in Outlet Shoppes of El Paso     1\n",
       "Former Mazda Dealership                            1\n",
       "Former E G Amish Furniture                         1\n",
       "Former F21                                         1\n",
       "Name: formerlocs, Length: 436, dtype: int64"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "formerlocsDF = pd.DataFrame()\n",
    "formerlocsDF['formerlocs'] = formerlocs\n",
    "formerlocsDF['formerlocs'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d98cfdb-956b-44d1-b064-1c78076a9f2b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}