DataCheckpointGroup096-Fa21.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COGS 108 - Data Checkpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Names\n",
    "\n",
    "- Joonsung Park\n",
    "- James Chung\n",
    "- Richard Gross\n",
    "- Madison Hambly\n",
    "- Colin Lintereur"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='research_question'></a>\n",
    "# Research Question"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do the most popular topics from previous COGS 108 final projects vary per quarter from 2019 to 2021? Also what other trends can we find in past submissions, like changes in word count, and number of graphs used?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Dataset Name: Past COGS108 final projects\n",
    "- Link to the dataset: [github.com/COGS108](github.com/COGS108)\n",
    "- Number of observations: 364\n",
    "\n",
    "We looked through the COGS108 github repositories and downloaded the zip file for each quarter of submissions we wanted to include in our analyis. From there, we process each jupyter notebook in the zip file and collect relevant data, parsing it to find the research question of the group. We then generate a csv file for each quarter, and once we have all the csv files of each quarter, we combine them into one final dataset we want to study. There are more quarters and submissions that are not yet included but we may include more if necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing libraries\n",
    "import numpy as np\n",
    "import json\n",
    "import zipfile\n",
    "import pandas as pd\n",
    "import re\n",
    "import paralleldots\n",
    "import seaborn as sns\n",
    "import glob\n",
    "\n",
    "# copy/paste API key for ParallelDots\n",
    "API_KEY = 'aSWrD1QSsUqdb8JYWJ6jMJMXrTxjFXKhdD4xQZKiUlQ'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(To avoid reaching the daily usage limit on the API, we each used our own API keys)\n",
    "\n",
    "We now search through the zip file of a quarter we want to study. We can get the quarter/group/year information from the file names. However, the research question is difficult to find since the formatting of every submission is not always the same. This may cause the file parser to not find any research question or find the wrong cell. However, this is very rare and does not actually ruin our analysis since all we need is a block of text that describes the topic of the project and any block of text near the research question tends to contain relevant information to the topic of the project. For now, even if the found text is not the research question, we will still process it (later we may exclude these projects, or handle them correctly)\n",
    "\n",
    "We manually ran this code for each project repo/zip in order to use different API keys. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting API key for classifier\n",
    "paralleldots.set_api_key(API_KEY)\n",
    "\n",
    "# initializing lists\n",
    "research_questions = {}\n",
    "qtr = []\n",
    "year = []\n",
    "group = []\n",
    "# special projects are marked with '_S' at the end of its name\n",
    "special = []\n",
    "word_counts = []\n",
    "index = 0\n",
    "\n",
    "# a magic number to guess if this string is the research question\n",
    "# (minimum character length)\n",
    "MIN_RQ_LEN = 30\n",
    "\n",
    "# reading zip file for a certain quarter\n",
    "with zipfile.ZipFile(\"zips/FinalProjects-Sp19-master.zip\", \"r\") as f:\n",
    "    for i, name in enumerate(f.namelist()):\n",
    "        # skip files that are not jupyter notebooks\n",
    "        if '.ipynb' not in name:\n",
    "            continue\n",
    "        rf = json.loads(f.read(name))\n",
    "        word_count = 0\n",
    "        rq_found = False\n",
    "        rq_in_header = False\n",
    "        \n",
    "        # loop through every cell\n",
    "        for j, cell in enumerate(rf['cells']):\n",
    "            if cell['cell_type'] == 'markdown':\n",
    "                # for the cell with the rq header, source_len > 1\n",
    "                # means the question is also in this cell\n",
    "                source_len = len(cell['source'])\n",
    "                for c in cell['source']:\n",
    "                    # if cell contains the research question header, update lists and extract the research question from the next cell\n",
    "                    if (rq_found == False) and (rq_in_header == True or re.search(r\"(^#.*research question)\", c.lower()) != None):\n",
    "                        # some other string in the cell's source list should be the question\n",
    "                        if source_len > 1 and rq_in_header == False:\n",
    "                            rq_in_header = True\n",
    "                            continue\n",
    "\n",
    "                        # if there is more text after the header, question may be in THIS cell\n",
    "                        if rq_in_header == True:\n",
    "                            # ignore '\\n' string or any string that is probably not the rq\n",
    "                            if len(c.lstrip()) < MIN_RQ_LEN:\n",
    "                                continue\n",
    "                            rq = str(c)\n",
    "                            rq_in_header = False\n",
    "                        # get the research question from the cell AFTER the cell that contains the header 'Research Question'\n",
    "                        else:\n",
    "                            rq = ' '.join(rf['cells'][j+1]['source'])\n",
    "                        # remove pesky html tags for API to handle correctly\n",
    "                        rq = re.sub('<[^<]+?>', '', rq)\n",
    "                        \n",
    "                        # extract quarter info from repo name\n",
    "                        date = re.search(r\"-(.*)-\", name).group(1)\n",
    "                        qtr.append(date[:2].upper())\n",
    "                        year.append(int(date[2:]))\n",
    "\n",
    "                        # extract group number\n",
    "                        fname = re.search(r\"/.*\", name).group(0)\n",
    "                        group.append(int(re.search(r\"[$0-9^]{2,3}\", fname)[0]))\n",
    "\n",
    "                        special.append(False if re.search(r\"_S\\.ipynb\", name) == None else True)\n",
    "                        \n",
    "                        research_questions[str(group[index])+\"_\"+qtr[index]+\"_\"+str(year[index])] = rq          \n",
    "                        rq_found = True\n",
    "                        index += 1\n",
    "                    word_count += len(c.replace('#', '').lstrip().split(' '))\n",
    "        if rq_found:\n",
    "            word_counts.append(word_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After collecting all the data we need from the submissions, we can use a text classifier to find the general topic of the submission. We are currently using an API made by ParallelDots to do this classification for us. See [here](https://apis.paralleldots.com/text_docs/index.html#taxonomy) for details on the API we are using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EXAMPLE:\n",
      " Can we emulate or improve on the effectivness of Open Comet for comet detection and mean olive moment extraction?  ::  TASTE\n"
     ]
    }
   ],
   "source": [
    "# initialize topics list\n",
    "topic = []\n",
    "\n",
    "# loop through research questions and find the most confident topic class from topic classifier\n",
    "for i, (g, q, y) in enumerate(zip(group, qtr, year)):\n",
    "    response = paralleldots.taxonomy(research_questions[str(group[i])+\"_\"+qtr[i]+\"_\"+str(year[i])])\n",
    "    topic.append(response['taxonomy'][0]['tag'])\n",
    "    if i == 0:\n",
    "        print(\"EXAMPLE:\\n\", research_questions[str(group[i])+\"_\"+qtr[i]+\"_\"+str(year[i])], \" :: \", topic[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have stored the topics of the submissions, we can run a sanity check to make sure that all of our list are the same length and then export the data to a csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50 50 50 50 50 50\n"
     ]
    }
   ],
   "source": [
    "# double check that all the lists are the same length\n",
    "print(len(group), len(qtr), len(year), len(topic), len(special), len(word_counts))\n",
    "\n",
    "# generate data frame and output it to a csv file\n",
    "d = {'Group':group, 'Quarter':qtr, 'Year':year, 'Topic':topic, 'Special':special, 'Word_Count':word_counts}\n",
    "df = pd.DataFrame(data = d)\n",
    "output = df.to_csv(path_or_buf=\"./csvs/\"+qtr[0] + \"20\" + str(year[0])+\".csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are collecting the data ourselves, there is not much work to be done cleaning the data after collecting all of the csv files for each quarter and merging them together into one large dataset. One column we do need to remove however is a column named 'Unnamed: 0'. This column is an artifact of reading csv files that store index information for each dataset. This column is redundant in our new dataset so we can remove it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observations: 364\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Group</th>\n",
       "      <th>Quarter</th>\n",
       "      <th>Year</th>\n",
       "      <th>Topic</th>\n",
       "      <th>Special</th>\n",
       "      <th>Word_Count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>FA</td>\n",
       "      <td>2020</td>\n",
       "      <td>ARTS &amp; CULTURE</td>\n",
       "      <td>False</td>\n",
       "      <td>7002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>FA</td>\n",
       "      <td>2020</td>\n",
       "      <td>BUSINESS</td>\n",
       "      <td>False</td>\n",
       "      <td>8689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>FA</td>\n",
       "      <td>2020</td>\n",
       "      <td>TECH</td>\n",
       "      <td>False</td>\n",
       "      <td>1864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>FA</td>\n",
       "      <td>2020</td>\n",
       "      <td>EDUCATION</td>\n",
       "      <td>False</td>\n",
       "      <td>5483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6</td>\n",
       "      <td>FA</td>\n",
       "      <td>2020</td>\n",
       "      <td>GREEN</td>\n",
       "      <td>False</td>\n",
       "      <td>4595</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Group Quarter  Year           Topic  Special  Word_Count\n",
       "0      1      FA  2020  ARTS & CULTURE    False        7002\n",
       "1      2      FA  2020        BUSINESS    False        8689\n",
       "2      4      FA  2020            TECH    False        1864\n",
       "3      5      FA  2020       EDUCATION    False        5483\n",
       "4      6      FA  2020           GREEN    False        4595"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using glob to get all csvs\n",
    "files = glob.glob('./csvs/*.csv')\n",
    "df_total = pd.DataFrame()\n",
    "entries = 0\n",
    "\n",
    "# read all csvs to dataframes and combine into one df\n",
    "for f_name in files:\n",
    "    csv = pd.read_csv(f_name)\n",
    "    df_total = df_total.append(csv)\n",
    "    entries = entries + len(csv.index)\n",
    "\n",
    "# remove artifact 'Unnamed'\n",
    "df_total = df_total.drop('Unnamed: 0', axis=1)\n",
    "\n",
    "assert(len(df_total.index) == entries)\n",
    "print('Observations: ' + str(len(df_total.index)))\n",
    "df_total.head()"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "04ac14935f6ed29b3349ee8f41114d2dfa2ba78ce87cf701ad9b7ca15955b787"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}