Part1C_Processing_Links.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "israeli-expression",
   "metadata": {},
   "source": [
    "# Project Overview\n",
    "In this notebook and subsequent notebooks I will attempt to predict mechanically meaningful labels associated with some fan-generated roleplaying game (RPG) content.\n",
    "\n",
    "I collected post and comment data from Reddit in multiple runs using a command-line tool I wrote for this purpose (see: [PRAW-CoDiaLS](https://github.com/nkuehnle/praw-codials))\n",
    "\n",
    "The actual text I will be attempting to predict is hosted on one of two websites: [GM Binder](https://gmbinder.com/) or [Homebrewery](https://homebrewery.naturalcrit.com/), which are tools/content hosting providers for fan-made RPG content, primarily Dungeons and Dragons 5th Edition (DnD5e).\n",
    "\n",
    "The labels make use of the user-assigned \"flair\" from the subreddit [/r/UnearthedArcana](https://www.reddit.com/r/UnearthedArcana/). UnearthedArcana describes itself as a source of \"homebrew\" (fan-generated) content and its community rules ensure that every post has a meaningful label (\"flair\") associated with it. These labels describe the \"type\" of game content that has been created, be it a character \"class\" (think achetypal high-fantasy characters knights, wizards, archers, etc) or a \"race\" (elves, dwarves, humans, etc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b84dc5c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Utility Imports\n",
    "import os\n",
    "from pathlib import Path\n",
    "# Imports for data processing/handling\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "# Custom modules\n",
    "from src.preprocessing.text_cleaning import clean_raw_scrapes\n",
    "from src.preprocessing import data_io"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7268caaf",
   "metadata": {},
   "source": [
    "# This Notebook\n",
    "Next, we'll import the submission data and collect the mixed html/css/markdown-style source code used by Homebrewery/GMBinder.\n",
    "\n",
    "**These will form the documents which we will later work to learn from.**\n",
    "\n",
    "I've defined several complex functions in src/dnd_scraper_tools.py unique to the websites that I'll be scraping.\n",
    "\n",
    "Accessing/calling/internals of these functions are decribed below.\n",
    "\n",
    "Note: This present notebook is revising a quick/scrappy project performed over a single weekend for a class, I already have collected the texts for many of the entries included. Some may need updating now that I've taken a more complete, clean approach. I'll integrate new data into the existing dataset. This will also form a blueprint for all future integration going forward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "independent-phrase",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a constant (CWD) to use\n",
    "CWD = Path(os.getcwd())\n",
    "DATA = CWD / 'data'\n",
    "PRAW = DATA / 'praw_data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "526f898b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw text files directory already exists\n",
      "Cleaned text files directory already exists\n"
     ]
    }
   ],
   "source": [
    "TEXT = DATA / 'text_files'\n",
    "try:\n",
    "    TEXT.mkdir(parents=True, exist_ok=False)\n",
    "    print(\"Raw text files directory created\")\n",
    "except:\n",
    "    print(\"Raw text files directory already exists\")\n",
    "\n",
    "CLEAN_TEXT = DATA / \"Clean_Text_Files\"\n",
    "try:\n",
    "    CLEAN_TEXT.mkdir(parents=True, exist_ok=False)\n",
    "    print(\"Cleaned text files directory created\")\n",
    "except:\n",
    "    print(\"Cleaned text files directory already exists\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5dbb6ab2",
   "metadata": {},
   "source": [
    "# Data Imports\n",
    "There are now five files I'll be working with:\n",
    "1. ProcessedData.csv\n",
    "2. CommentsFiltered.csv\n",
    "3. SubmissionsFiltered.csv\n",
    "\n",
    "If you are not familiar with Reddit, when users post it can take two forms, \"Submissions\" and \"comments.\"\n",
    "\n",
    "Submissions are associated with a specific community or \"subreddit\" and users can comment on Submissions to discuss them or provide additional commentary.\n",
    "Submissions can come in multiple forms: link, text, image, etc. The submissions here are only link submissions. Comments are searched for all other types of submissions as users will frequently post an image version of their content and then link to the text copy in the comments on these subreddits. The vast majority of content on /r/UnearthedArcana comes in this second form (an image submission with the link in the comments).\n",
    "\n",
    "In any case, PRAW-CoDiaLS is written so that they both share the same overall structure, although certain fields are only used for comments (and blank otherwise).\n",
    "\n",
    "\n",
    "### Shared Fields\n",
    "\n",
    "| column name | description |\n",
    "| ----- | ----- |\n",
    "| link | Raw text URL |\n",
    "| submission_author | Reddit user who posted the comment |\n",
    "| submission_id | A unique ID used for locating the submission on Reddit |\n",
    "| submission_title | The title of the submission on Reddit |\n",
    "| subreddit | The name of the Subreddit the content was found on (in this case, either UnearthedArcana or DnDHomebrew) |\n",
    "| submission_flair | the figure number as indicated in the manuscript |\n",
    "| submission_score | The approximate number of net positive votes received by the submission on Reddit, the actual values are obfuscated by Reddit to avoid manipulation |\n",
    "| submission_upvote_ratio | The ratio of positive to negative votes |\n",
    "| submission_date | Date on which the submission was originally made |\n",
    "\n",
    "### Comment-only fields\n",
    "\n",
    "| column name | description |\n",
    "| ----- | ----- |\n",
    "| comment_author | Reddit user who posted the comment |\n",
    "| comment_id | A unique ID used for locating the comment on Reddit |\n",
    "| comment_score | The approximate number of net positive votes received by the comment on Reddit, the actual values are obfuscated by Reddit to avoid manipulation |\n",
    "| comment_body | Text of the comment containing markdown elements |\n",
    "| comment_date | Date on which the comment was originally posted |\n",
    "| related_link | Boolean indicator for whether linked text was related to the labeled post (MultiLinkCmts.csv only) |\n",
    "| manually_reviewed | Boolean indicator of whether the linked text has been manually reviewed for accuracy |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "deluxe-heart",
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_subs = pd.read_csv(PRAW/'SubmissionsFiltered.csv', index_col='idx')\n",
    "filtered_cmts = pd.read_csv(PRAW/'FinalCommentsFiltered.csv', index_col='idx')\n",
    "dropped_subs = pd.read_csv(PRAW/'SubmissionsDropped.csv', index_col='idx')\n",
    "dropped_cmts = pd.read_csv(PRAW/'FinalCommentsDropped.csv', index_col='idx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "159df875",
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df = pd.concat([filtered_subs,filtered_cmts])\n",
    "dropped_df = pd.concat([dropped_subs,dropped_cmts])\n",
    "collected_df = pd.read_csv(PRAW/'CollectedData.csv')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1c537e60",
   "metadata": {},
   "source": [
    "## Updating manually-reviewed labels\n",
    "Some of the labels were manually reviewed and then updated in the previous steps. We'll fix that now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "070a8f43",
   "metadata": {},
   "outputs": [],
   "source": [
    "manually_updated = filtered_df[filtered_df['manually_reviewed'] == True].copy()\n",
    "collected_df = collected_df.sort_values(by='link')\n",
    "manually_updated = manually_updated.sort_values(by='link')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "59909673",
   "metadata": {},
   "outputs": [],
   "source": [
    "to_update = collected_df['link'].isin(manually_updated['link'])\n",
    "new_vals = manually_updated[manually_updated['link'].isin(collected_df['link'])]\n",
    "mapper = new_vals.set_index(\"link\")[\"submission_flair\"].to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "66293de3",
   "metadata": {},
   "outputs": [],
   "source": [
    "collected_df.loc[to_update, 'submission_flair'] = collected_df.loc[to_update, 'link'].map(mapper)\n",
    "collected_df = collected_df.sort_values(by='UID')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "12b8898f",
   "metadata": {},
   "source": [
    "## Inspecting New Data\n",
    "\n",
    "Here I'll do the following:\n",
    "\n",
    "1. Check how much of the data is new (not already in the collected dataset\n",
    "2. Check if any previously collected data has been excluded by my newer, more nuanced filtering approach\n",
    "3. If any previously collected links have now been included, I'll take a look at why and assess whether their exclusion was appropriate or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "17c3cc1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2356 of 3376 filtered links have been collected already.\n",
      "452 of 1345 dropped links have been collected already.\n"
     ]
    }
   ],
   "source": [
    "collected_filtered = filtered_df['link'].isin(collected_df['link'])\n",
    "collected_dropped = dropped_df['link'].isin(collected_df['link'])\n",
    "print(f\"{sum(collected_filtered)} of {len(filtered_df)} filtered links have been collected already.\")\n",
    "print(f\"{sum(collected_dropped)} of {len(dropped_df)} dropped links have been collected already.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b34f04f9",
   "metadata": {},
   "source": [
    "Based on the above, we can see that about 3/4 of the links have already had text collected. However, the new links should nicely expand the corpus of texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a98610bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7 links that are strictly in the discard collection (10 entries) have already been collected.\n"
     ]
    }
   ],
   "source": [
    "collected_dropped_only = collected_dropped & (~dropped_df['link'].isin(filtered_df['link']))\n",
    "collected_dropped_only = dropped_df[collected_dropped_only]\n",
    "print(f\"{len(collected_dropped_only['link'].unique())} links that are strictly in the discard collection ({len(collected_dropped_only)} entries) have already been collected.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d4e17cb2",
   "metadata": {},
   "source": [
    "Based on the above, we can see that only a few of the links that we've already collected were dropped and aren't simply duplicates of links found in the filtered dataset. Now I'll see why these links were excluded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b2617290",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Invalid/Missing Content Type    10\n",
       "Name: discard_reason, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collected_dropped_only['discard_reason'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1c375cc5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['https://homebrewery.naturalcrit.com/share/B1Hju_QaTx',\n",
       "       'https://gmbinder.com/share/-Kx5i9fuxhuci6BUrbJq',\n",
       "       'https://gmbinder.com/share/-MbAAe7gUkgvdAdE47qp',\n",
       "       'https://gmbinder.com/share/-LsEd761uB8mQlHJmpYd',\n",
       "       'https://gmbinder.com/share/-LM5jFKi0XzDEt-KZACa',\n",
       "       'https://gmbinder.com/share/-LNKsc1ZWIwwDRvfScQu',\n",
       "       'https://gmbinder.com/share/-MVT7DxS1Z08371vBCcL'], dtype=object)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manually_inspected_links = collected_dropped_only['link'].unique()\n",
    "manually_inspected_links"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6ac09308",
   "metadata": {},
   "source": [
    "Manual inspection confirms that none of these links contain useful training data."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "53bfbb75",
   "metadata": {},
   "source": [
    "#### Drop Discarded Links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7c5881f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "collected_df = collected_df[~collected_df['link'].isin(manually_inspected_links)]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "352e23df",
   "metadata": {},
   "source": [
    "#### Are any collected links not in the filtered dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "806b7fbd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12 links have been collected but are not contained in the present dataset\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UID</th>\n",
       "      <th>link</th>\n",
       "      <th>submission_author</th>\n",
       "      <th>submission_id</th>\n",
       "      <th>submission_title</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>submission_flair</th>\n",
       "      <th>submission_score</th>\n",
       "      <th>submission_upvote_ratio</th>\n",
       "      <th>submission_date</th>\n",
       "      <th>comment_author</th>\n",
       "      <th>comment_id</th>\n",
       "      <th>comment_score</th>\n",
       "      <th>comment_body</th>\n",
       "      <th>comment_date</th>\n",
       "      <th>src_url</th>\n",
       "      <th>manually_reviewed</th>\n",
       "      <th>related_link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>127</td>\n",
       "      <td>https://gmbinder.com/images/TdynXtN.png</td>\n",
       "      <td>RSquared</td>\n",
       "      <td>o8jg66</td>\n",
       "      <td>Wild Magic, Revisited (v1.4): When you have a ...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>119</td>\n",
       "      <td>0.96</td>\n",
       "      <td>1624742801</td>\n",
       "      <td>RSquared</td>\n",
       "      <td>h35bgcy</td>\n",
       "      <td>5.0</td>\n",
       "      <td>## Wild Magic, Revisited\\nPDF/GMBinder: [https...</td>\n",
       "      <td>1624743040</td>\n",
       "      <td>https://www.gmbinder.com/images/TdynXtN.png</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>174</td>\n",
       "      <td>https://gmbinder.com/share/-Mo3iAw7pM6ey8vc3U5o</td>\n",
       "      <td>vonBoomslang</td>\n",
       "      <td>s9d5bl</td>\n",
       "      <td>[Subclass] Fighter: Swiftblade</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>74</td>\n",
       "      <td>0.97</td>\n",
       "      <td>NaN</td>\n",
       "      <td>vonBoomslang</td>\n",
       "      <td>htlw4sf</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[Art](https://www.artstation.com/artwork/DBWEe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.gmbinder.com/share/-Mo3iAw7pM6ey8v...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>326</td>\n",
       "      <td>https://gmbinder.com/share/-LTuQ9tRb_ZGFIfKLno4</td>\n",
       "      <td>FungalBrews</td>\n",
       "      <td>stvbuc</td>\n",
       "      <td>The Warrior's Codex - a 156-page remaster of a...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Compendium</td>\n",
       "      <td>280</td>\n",
       "      <td>0.98</td>\n",
       "      <td>2022-02-16 13:00:13</td>\n",
       "      <td>FungalBrews</td>\n",
       "      <td>hx75s6n</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Can't believe I forgot to put the [GMBinder](h...</td>\n",
       "      <td>2022-02-16 17:39:09</td>\n",
       "      <td>https://www.gmbinder.com/share/-LTuQ9tRb_ZGFIf...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>384</th>\n",
       "      <td>391</td>\n",
       "      <td>https://gmbinder.com/share/-MZ8TgPaZ7Cy-GfA6ODa</td>\n",
       "      <td>zaelos_3</td>\n",
       "      <td>t08upg</td>\n",
       "      <td>Class: Warlord by zaelos_3 (v1.8.4) - Non-magi...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Class</td>\n",
       "      <td>50</td>\n",
       "      <td>0.90</td>\n",
       "      <td>2022-02-24 11:32:53</td>\n",
       "      <td>zaelos_3</td>\n",
       "      <td>hy8as2c</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Hey! Still working on adjustments, but in gene...</td>\n",
       "      <td>2022-02-24 11:52:14</td>\n",
       "      <td>https://www.gmbinder.com/share/-MZ8TgPaZ7Cy-Gf...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>435</th>\n",
       "      <td>444</td>\n",
       "      <td>https://gmbinder.com/images/NwecXt3.png</td>\n",
       "      <td>Frail_Luna</td>\n",
       "      <td>lx0mo3</td>\n",
       "      <td>Way of the Dance - A Monastic Tradition for dr...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>1282</td>\n",
       "      <td>0.99</td>\n",
       "      <td>1614795693</td>\n",
       "      <td>Nihil_esque</td>\n",
       "      <td>gpmg6gf</td>\n",
       "      <td>2.0</td>\n",
       "      <td>I don't know how they achieved it, but you can...</td>\n",
       "      <td>1614835599</td>\n",
       "      <td>https://www.gmbinder.com/images/NwecXt3.png</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>463</th>\n",
       "      <td>472</td>\n",
       "      <td>https://gmbinder.com/share/-Muk72f9o48Y_sxYWEsD</td>\n",
       "      <td>Sensitive_Coyote_865</td>\n",
       "      <td>shyemc</td>\n",
       "      <td>Summon weapon (corrected) - A cantrip</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Spell</td>\n",
       "      <td>513</td>\n",
       "      <td>0.97</td>\n",
       "      <td>2022-02-01 16:03:05</td>\n",
       "      <td>Sensitive_Coyote_865</td>\n",
       "      <td>hv5ay6b</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Here is a simple conjuration cantrip I designe...</td>\n",
       "      <td>2022-02-01 16:04:44</td>\n",
       "      <td>https://www.gmbinder.com/share/-Muk72f9o48Y_sx...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>622</th>\n",
       "      <td>635</td>\n",
       "      <td>https://gmbinder.com/share/-MY3BKQPmwzGOm8b58d1</td>\n",
       "      <td>KaiTries5</td>\n",
       "      <td>npyqnp</td>\n",
       "      <td>The Chevalier 2.0 - A Charismatic Martial Clas...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Class</td>\n",
       "      <td>349</td>\n",
       "      <td>0.97</td>\n",
       "      <td>2021-06-01 16:59:03</td>\n",
       "      <td>KaiTries5</td>\n",
       "      <td>h07o3fm</td>\n",
       "      <td>16.0</td>\n",
       "      <td>Hello, r/UnearthedArcana!\\n\\nA little over a w...</td>\n",
       "      <td>2021-06-01 17:02:47</td>\n",
       "      <td>https://www.gmbinder.com/share/-MY3BKQPmwzGOm8...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1248</th>\n",
       "      <td>1659</td>\n",
       "      <td>https://gmbinder.com/share/-MsDn-S30J-qAuY7b0VU</td>\n",
       "      <td>FriskyRisque</td>\n",
       "      <td>siukf3</td>\n",
       "      <td>The Seeker class is updated! [Alpha v1.0] Tell...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Class</td>\n",
       "      <td>254</td>\n",
       "      <td>0.97</td>\n",
       "      <td>2022-02-02 17:34:39</td>\n",
       "      <td>FriskyRisque</td>\n",
       "      <td>hvavqh9</td>\n",
       "      <td>5.0</td>\n",
       "      <td>[Google PDF](https://drive.google.com/file/d/1...</td>\n",
       "      <td>2022-02-02 17:35:11</td>\n",
       "      <td>https://www.gmbinder.com/share/-MsDn-S30J-qAuY...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1304</th>\n",
       "      <td>1716</td>\n",
       "      <td>https://gmbinder.com/share/-LtakOF3PZ1Re19Sn6ca</td>\n",
       "      <td>actlikeyoubelong__c</td>\n",
       "      <td>nghrwf</td>\n",
       "      <td>Oath of the Frontier: The frontier is vast and...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>510</td>\n",
       "      <td>0.98</td>\n",
       "      <td>2021-05-19 21:31:08</td>\n",
       "      <td>actlikeyoubelong__c</td>\n",
       "      <td>gyvyyxm</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Swift justice has been changed to quick draw! ...</td>\n",
       "      <td>2021-05-21 00:15:47</td>\n",
       "      <td>https://www.gmbinder.com/share/-LtakOF3PZ1Re19...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1368</th>\n",
       "      <td>1782</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/v_5s...</td>\n",
       "      <td>Xenoezen</td>\n",
       "      <td>saxugc</td>\n",
       "      <td>Eldritch Invocation: Pact Tactics</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Feat</td>\n",
       "      <td>1438</td>\n",
       "      <td>0.97</td>\n",
       "      <td>2022-01-23 16:48:55</td>\n",
       "      <td>Xenoezen</td>\n",
       "      <td>htwcgso</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Here's an eldritch invocation for chainlocks, ...</td>\n",
       "      <td>2022-01-23 16:50:49</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/v_5...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1662</th>\n",
       "      <td>2117</td>\n",
       "      <td>https://gmbinder.com/images/3ORYq56.png</td>\n",
       "      <td>KibblesTasty</td>\n",
       "      <td>jbnfms</td>\n",
       "      <td>Kibbles Battle System - A campaign-spanning De...</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>5e</td>\n",
       "      <td>919</td>\n",
       "      <td>0.98</td>\n",
       "      <td>1602768136</td>\n",
       "      <td>KibblesTasty</td>\n",
       "      <td>g8w9qr4</td>\n",
       "      <td>29.0</td>\n",
       "      <td>### What is this?\\n\\nThis is a framework that ...</td>\n",
       "      <td>1602768165</td>\n",
       "      <td>https://www.gmbinder.com/images/3ORYq56.png</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2228</th>\n",
       "      <td>2811</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/share/1qJm...</td>\n",
       "      <td>SwampertSpawner</td>\n",
       "      <td>n7ist3</td>\n",
       "      <td>This died in new, so I'm want to show once aga...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Subclass</td>\n",
       "      <td>101</td>\n",
       "      <td>0.90</td>\n",
       "      <td>NaN</td>\n",
       "      <td>SwampertSpawner</td>\n",
       "      <td>gxd1heo</td>\n",
       "      <td>4.0</td>\n",
       "      <td>*Don't you want to taste the poisonous freedom...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://homebrewery.naturalcrit.com/source/1qJ...</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       UID                                               link  \\\n",
       "123    127            https://gmbinder.com/images/TdynXtN.png   \n",
       "169    174    https://gmbinder.com/share/-Mo3iAw7pM6ey8vc3U5o   \n",
       "319    326    https://gmbinder.com/share/-LTuQ9tRb_ZGFIfKLno4   \n",
       "384    391    https://gmbinder.com/share/-MZ8TgPaZ7Cy-GfA6ODa   \n",
       "435    444            https://gmbinder.com/images/NwecXt3.png   \n",
       "463    472    https://gmbinder.com/share/-Muk72f9o48Y_sxYWEsD   \n",
       "622    635    https://gmbinder.com/share/-MY3BKQPmwzGOm8b58d1   \n",
       "1248  1659    https://gmbinder.com/share/-MsDn-S30J-qAuY7b0VU   \n",
       "1304  1716    https://gmbinder.com/share/-LtakOF3PZ1Re19Sn6ca   \n",
       "1368  1782  https://homebrewery.naturalcrit.com/share/v_5s...   \n",
       "1662  2117            https://gmbinder.com/images/3ORYq56.png   \n",
       "2228  2811  https://homebrewery.naturalcrit.com/share/1qJm...   \n",
       "\n",
       "         submission_author submission_id  \\\n",
       "123               RSquared        o8jg66   \n",
       "169           vonBoomslang        s9d5bl   \n",
       "319            FungalBrews        stvbuc   \n",
       "384               zaelos_3        t08upg   \n",
       "435             Frail_Luna        lx0mo3   \n",
       "463   Sensitive_Coyote_865        shyemc   \n",
       "622              KaiTries5        npyqnp   \n",
       "1248          FriskyRisque        siukf3   \n",
       "1304   actlikeyoubelong__c        nghrwf   \n",
       "1368              Xenoezen        saxugc   \n",
       "1662          KibblesTasty        jbnfms   \n",
       "2228       SwampertSpawner        n7ist3   \n",
       "\n",
       "                                       submission_title        subreddit  \\\n",
       "123   Wild Magic, Revisited (v1.4): When you have a ...  UnearthedArcana   \n",
       "169                      [Subclass] Fighter: Swiftblade      DnDHomebrew   \n",
       "319   The Warrior's Codex - a 156-page remaster of a...  UnearthedArcana   \n",
       "384   Class: Warlord by zaelos_3 (v1.8.4) - Non-magi...  UnearthedArcana   \n",
       "435   Way of the Dance - A Monastic Tradition for dr...  UnearthedArcana   \n",
       "463               Summon weapon (corrected) - A cantrip  UnearthedArcana   \n",
       "622   The Chevalier 2.0 - A Charismatic Martial Clas...  UnearthedArcana   \n",
       "1248  The Seeker class is updated! [Alpha v1.0] Tell...  UnearthedArcana   \n",
       "1304  Oath of the Frontier: The frontier is vast and...  UnearthedArcana   \n",
       "1368                  Eldritch Invocation: Pact Tactics  UnearthedArcana   \n",
       "1662  Kibbles Battle System - A campaign-spanning De...      DnDHomebrew   \n",
       "2228  This died in new, so I'm want to show once aga...  UnearthedArcana   \n",
       "\n",
       "     submission_flair  submission_score  submission_upvote_ratio  \\\n",
       "123          Subclass               119                     0.96   \n",
       "169          Subclass                74                     0.97   \n",
       "319        Compendium               280                     0.98   \n",
       "384             Class                50                     0.90   \n",
       "435          Subclass              1282                     0.99   \n",
       "463             Spell               513                     0.97   \n",
       "622             Class               349                     0.97   \n",
       "1248            Class               254                     0.97   \n",
       "1304         Subclass               510                     0.98   \n",
       "1368             Feat              1438                     0.97   \n",
       "1662               5e               919                     0.98   \n",
       "2228         Subclass               101                     0.90   \n",
       "\n",
       "          submission_date        comment_author comment_id  comment_score  \\\n",
       "123            1624742801              RSquared    h35bgcy            5.0   \n",
       "169                   NaN          vonBoomslang    htlw4sf            2.0   \n",
       "319   2022-02-16 13:00:13           FungalBrews    hx75s6n            1.0   \n",
       "384   2022-02-24 11:32:53              zaelos_3    hy8as2c            2.0   \n",
       "435            1614795693           Nihil_esque    gpmg6gf            2.0   \n",
       "463   2022-02-01 16:03:05  Sensitive_Coyote_865    hv5ay6b            1.0   \n",
       "622   2021-06-01 16:59:03             KaiTries5    h07o3fm           16.0   \n",
       "1248  2022-02-02 17:34:39          FriskyRisque    hvavqh9            5.0   \n",
       "1304  2021-05-19 21:31:08   actlikeyoubelong__c    gyvyyxm            1.0   \n",
       "1368  2022-01-23 16:48:55              Xenoezen    htwcgso           15.0   \n",
       "1662           1602768136          KibblesTasty    g8w9qr4           29.0   \n",
       "2228                  NaN       SwampertSpawner    gxd1heo            4.0   \n",
       "\n",
       "                                           comment_body         comment_date  \\\n",
       "123   ## Wild Magic, Revisited\\nPDF/GMBinder: [https...           1624743040   \n",
       "169   [Art](https://www.artstation.com/artwork/DBWEe...                  NaN   \n",
       "319   Can't believe I forgot to put the [GMBinder](h...  2022-02-16 17:39:09   \n",
       "384   Hey! Still working on adjustments, but in gene...  2022-02-24 11:52:14   \n",
       "435   I don't know how they achieved it, but you can...           1614835599   \n",
       "463   Here is a simple conjuration cantrip I designe...  2022-02-01 16:04:44   \n",
       "622   Hello, r/UnearthedArcana!\\n\\nA little over a w...  2021-06-01 17:02:47   \n",
       "1248  [Google PDF](https://drive.google.com/file/d/1...  2022-02-02 17:35:11   \n",
       "1304  Swift justice has been changed to quick draw! ...  2021-05-21 00:15:47   \n",
       "1368  Here's an eldritch invocation for chainlocks, ...  2022-01-23 16:50:49   \n",
       "1662  ### What is this?\\n\\nThis is a framework that ...           1602768165   \n",
       "2228  *Don't you want to taste the poisonous freedom...                  NaN   \n",
       "\n",
       "                                                src_url manually_reviewed  \\\n",
       "123         https://www.gmbinder.com/images/TdynXtN.png               NaN   \n",
       "169   https://www.gmbinder.com/share/-Mo3iAw7pM6ey8v...               NaN   \n",
       "319   https://www.gmbinder.com/share/-LTuQ9tRb_ZGFIf...               NaN   \n",
       "384   https://www.gmbinder.com/share/-MZ8TgPaZ7Cy-Gf...               NaN   \n",
       "435         https://www.gmbinder.com/images/NwecXt3.png               NaN   \n",
       "463   https://www.gmbinder.com/share/-Muk72f9o48Y_sx...               NaN   \n",
       "622   https://www.gmbinder.com/share/-MY3BKQPmwzGOm8...               NaN   \n",
       "1248  https://www.gmbinder.com/share/-MsDn-S30J-qAuY...               NaN   \n",
       "1304  https://www.gmbinder.com/share/-LtakOF3PZ1Re19...               NaN   \n",
       "1368  https://homebrewery.naturalcrit.com/source/v_5...               NaN   \n",
       "1662        https://www.gmbinder.com/images/3ORYq56.png               NaN   \n",
       "2228  https://homebrewery.naturalcrit.com/source/1qJ...              True   \n",
       "\n",
       "     related_link  \n",
       "123           NaN  \n",
       "169           NaN  \n",
       "319           NaN  \n",
       "384           NaN  \n",
       "435           NaN  \n",
       "463           NaN  \n",
       "622           NaN  \n",
       "1248          NaN  \n",
       "1304          NaN  \n",
       "1368          NaN  \n",
       "1662          NaN  \n",
       "2228          NaN  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collected_only = collected_df[~collected_df['link'].isin(filtered_df['link'])]\n",
    "print(f\"{len(collected_only)} links have been collected but are not contained in the present dataset\")\n",
    "collected_only.to_csv(PRAW/'MissingCollected.csv', index_label='idx')\n",
    "collected_only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6dc639c7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['https://gmbinder.com/images/TdynXtN.png',\n",
       "       'https://gmbinder.com/share/-Mo3iAw7pM6ey8vc3U5o',\n",
       "       'https://gmbinder.com/share/-LTuQ9tRb_ZGFIfKLno4',\n",
       "       'https://gmbinder.com/share/-MZ8TgPaZ7Cy-GfA6ODa',\n",
       "       'https://gmbinder.com/images/NwecXt3.png',\n",
       "       'https://gmbinder.com/share/-Muk72f9o48Y_sxYWEsD',\n",
       "       'https://gmbinder.com/share/-MY3BKQPmwzGOm8b58d1',\n",
       "       'https://gmbinder.com/share/-MsDn-S30J-qAuY7b0VU',\n",
       "       'https://gmbinder.com/share/-LtakOF3PZ1Re19Sn6ca',\n",
       "       'https://homebrewery.naturalcrit.com/share/v_5saUNTTEtN]',\n",
       "       'https://gmbinder.com/images/3ORYq56.png',\n",
       "       'https://homebrewery.naturalcrit.com/share/1qJmpQuXYOOUZErtdKwC'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collected_only['link'].unique()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d87dcbeb",
   "metadata": {},
   "source": [
    "#### UID 127, 444, 2117, and 2811 are bad links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5826f3ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "collected_df = collected_df[~collected_df['UID'].isin([127,444,2117,2811])].copy()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ad95992d",
   "metadata": {},
   "source": [
    "# Collect Markdown/Source Code\n",
    "\n",
    "The code below attempts to scrape the actual text for each entry. This code only runs if the the data hasn't been processed already."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "457ba8a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1020 links not in existing collection\n"
     ]
    }
   ],
   "source": [
    "new_links = filtered_df[~filtered_df['link'].isin(collected_df['link'])]\n",
    "has_new_links = len(new_links) >= 1\n",
    "print(f\"{len(new_links)} links not in existing collection\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "21788c67-1aa9-4587-bdde-d81476e5afee",
   "metadata": {},
   "source": [
    "## dnd_scraper_tools\n",
    "\n",
    "As mentioned previously, the actual text scraping is handled by functions defined in dnd_scraper_tools.\n",
    "\n",
    "Note that it's possible for either gmbinder.com or homebrewery.naturalcrit.com to change in ways substantially enough that these functions would be broken in the future.\n",
    "\n",
    "### `get_source_texts()`\n",
    "General function: his function wraps three other functions.\n",
    "\n",
    "It also includes rate_limit and pass_attempts variables that can be used to slow down the rate of requests that are sent to these sites to minimize impact(s).\n",
    "\n",
    "In the event that any requests fail due to HTTP exceptions, the function will pass back over those URLs a number of times equal to pass_attempts.\n",
    "\n",
    "#### Step 1: Find the best URL to get the markdown text.\n",
    "\n",
    "The first function simply checks for a \"source code' button on the rendered view linked to be each submission (typically users link to this view as it is more aesthetically pleasing and the primary draw of these websites).\n",
    "\n",
    "This clean raw source code is a form of Markdown (and potentially some CSS) unique to GMBinder/Homebrewery and designed for rendering nice content. Server-side, this is converted to a much more complex mix of HTML/JS/CSS that is delivered to the clients to produce the rendered viewing page.\n",
    "\n",
    "In the event that there is a raw source page, we'll try to use that as it's generally going to provide clear section delineations\n",
    "\n",
    "If the source page is not found, we'll try to collect the text from the rendered page and convert it to markdown, so in these cases, we'll return the original URL.\n",
    "\n",
    "Note: This works only for GMBinder's rendered pages, as Homebrewery uses a ReactJS framework that doesn't play nicely with BeautifulSoup4. To keep things simple, the function returns a value of None if the source is not available for a Homebrewery link. We'll need to drop any such rows later.\n",
    "\n",
    "From what I observed clicking through some of the links, an enabled source link seems to be standard on Homebrewery, but can be optionally disabled on GMBinder (and is done with appreciable frequency).\n",
    "\n",
    "#### Step 2: Collect Text\n",
    "\n",
    "The second function from the dnd_scraper_tools module is used to collect the text from the newly located source URLs.\n",
    "\n",
    "This can either be the raw markdown source text or the HTML/JS/CSS mixture from the rendered view.\n",
    "\n",
    "In either case, we will try to make sure it's in the form of markdown, not HTML or raw text.\n",
    "\n",
    "There are still many undesirable elements at this point, but we'll clean these up towards the end of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "78f053c4-0c32-440a-98ad-80e9222a1b20",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data\n",
      "All target data has been collected already.\n"
     ]
    }
   ],
   "source": [
    "new_collection_path = PRAW/'NewCollectionFiltered.csv'\n",
    "TRY_N_TIMES = 7\n",
    "if has_new_links:\n",
    "    from src import get_source_texts\n",
    "    \n",
    "    new_links = new_links.reset_index(drop=False).copy()\n",
    "\n",
    "    if new_collection_path.is_file():\n",
    "        print(\"Loading data\")\n",
    "        _new_links = pd.read_csv(new_collection_path)\n",
    "        start_range = max([collected_df['UID'].max(), _new_links['UID'].max()]) + 1\n",
    "        missing_links = new_links[~new_links['link'].isin(_new_links['link'])].copy()\n",
    "        missing_links.insert(0, 'UID', range(start_range, start_range + len(missing_links)))\n",
    "        new_links = _new_links[_new_links['link'].isin(new_links['link'])].copy()\n",
    "\n",
    "        mapper = manually_updated.set_index(\"link\")['submission_flair'].to_dict()\n",
    "        new_links.loc[new_links['link'].isin(mapper.keys()),'submission_flair'] = new_links[new_links['link'].isin(mapper.keys())]['link'].map(mapper)\n",
    "        new_links.loc[new_links['link'].isin(mapper.keys()), 'manually_reviewed'] = True\n",
    "        \n",
    "        if len(missing_links) > 0:\n",
    "            print(f\"Starting new entries at UID {start_range}\")\n",
    "            missing_links = get_source_texts(\n",
    "                df = missing_links,\n",
    "                rate_limit = 1, # Max one request per second\n",
    "                pass_attempts = TRY_N_TIMES, # Try up to N times for timeout, etc,\n",
    "                verbose = True\n",
    "                )\n",
    "            new_links = pd.concat([_new_links, missing_links])\n",
    "            new_links.to_csv(new_collection_path, index=False)\n",
    "        else:\n",
    "            new_links.to_csv(new_collection_path, index=False)\n",
    "            print(\"All target data has been collected already.\")\n",
    "    else:\n",
    "        start_range = collected_df['UID'].max() + 1\n",
    "        new_links.insert(0, 'UID', range(start_range, start_range + len(new_links)))\n",
    "        # Get new texts\n",
    "        new_links = get_source_texts(\n",
    "            df = new_links,\n",
    "            rate_limit = 1, # Max one request per second\n",
    "            pass_attempts = TRY_N_TIMES, # Try up to N times for timeout, etc,\n",
    "            verbose = True\n",
    "            )\n",
    "\n",
    "        new_links.to_csv(new_collection_path, index=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0289b390-dd85-46d2-9c57-1c0f6ab1b85b",
   "metadata": {},
   "source": [
    "Some of the links are to dead/removed content or content that is purely an image (typically from a single user). I didn't plan for this in my initial collection, so I'm going to drop them now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "32dae69b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://gmbinder.com/share/-LlMLqn89zSK9HarUVb0\n",
      "https://homebrewery.naturalcrit.com/share/fxIFF3_otodv\n",
      "https://homebrewery.naturalcrit.com/share/rkv7KcWhN\n",
      "https://homebrewery.naturalcrit.com/share/S1bgDI6oe\n",
      "https://homebrewery.naturalcrit.com/share/SJb7f4h2Jm\n",
      "https://homebrewery.naturalcrit.com/share/Q8h-5qPRnSIt\n",
      "https://homebrewery.naturalcrit.com/share/AIs9CFcEl5cZ\n",
      "https://gmbinder.com/share/-MpmR3BmlaL98mXF9Tat\n",
      "https://gmbinder.com/share/-M6Pbsi5tQ896CKv0i6\n",
      "https://gmbinder.com/share/-MZK9zMAViuPzPOGJwLj\n",
      "https://homebrewery.naturalcrit.com/share/rJbFLYlib\n",
      "https://homebrewery.naturalcrit.com/share/vlLpF0uS968g\n",
      "https://homebrewery.naturalcrit.com/share/BJQGzT-md4\n",
      "https://homebrewery.naturalcrit.com/share/r1Z_Pk2a1Q\n",
      "https://gmbinder.com/share/-LjvXYwjrv22ExcBsPGV\n",
      "https://homebrewery.naturalcrit.com/share/aFjGJsE89\n",
      "https://homebrewery.naturalcrit.com/share/Lk4tshmkY0en\n",
      "https://homebrewery.naturalcrit.com/share/F3gBoje3Lg45\n",
      "https://homebrewery.naturalcrit.com/share/B1m1TTXHBb\n",
      "https://homebrewery.naturalcrit.com/share/2B_HuyWQoqZ0\n",
      "https://homebrewery.naturalcrit.com/share/Sk7h4hjchE\n",
      "https://gmbinder.com/share/-L6eThyjH-OkFV7BJ0xG\n",
      "https://gmbinder.com\n",
      "https://homebrewery.naturalcrit.com/share/Zt8-J3TWCG3n\n",
      "https://gmbinder.com/share/-Mvw2gMRlTADo5_u5R9s\n",
      "https://gmbinder.com/share/-M_LB4An6ri5jmnd1qza\n",
      "https://gmbinder.com/share/-MtnpxGqdvmlswLd27Oz\n",
      "https://gmbinder.com/share/-Mty0eeUScasOmzY_TMC\n",
      "https://gmbinder.com/share/-N7SwHuOcilHzClx-d-j\n",
      "https://homebrewery.naturalcrit.com/share/1YKPzuic_oqR74vbPOopM1hnVWqDl6Qc0lH1ynRaq3HMq\n",
      "https://gmbinder.com/share/-N8VPGOo5mjA9kZ6roNk\n",
      "https://homebrewery.naturalcrit.com\n",
      "https://homebrewery.naturalcrit.com/share/5XsddLUwKw6Q\n",
      "https://homebrewery.naturalcrit.com/share/031XdzxlPsJ3\n",
      "https://homebrewery.naturalcrit.com/share/HWw_pO0DJZl0\n",
      "https://homebrewery.naturalcrit.com/share/jZVcO_txqe8n\n",
      "https://homebrewery.naturalcrit.com/share/1AV0SDXaKU9wG24g4iFZr3j5-9nYG5H87vDct4GxYB3HX\n",
      "https://homebrewery.naturalcrit.com/share/U8cR7MacUr6k\n",
      "https://homebrewery.naturalcrit.com/share/HyYtpvzp4\n",
      "https://gmbinder.com/share/-M3dLiCm9SDlu-WiQVfA/-MRuamQvXboTFSzS_IcJ\n",
      "https://homebrewery.naturalcrit.com/share/sI4EIPWeU\n",
      "https://gmbinder.com/share/-MTyHwf6Opi06U7qCOKg\n",
      "https://gmbinder.com/share/-MUAIwhAwockCRv10XGI\n",
      "https://homebrewery.naturalcrit.com/share/h5u7Lb150\n",
      "https://homebrewery.naturalcrit.com/share/mIzeXcQcYi2G\n",
      "https://homebrewery.naturalcrit.com/share/1aPDl8cubz8nyCwR8izuC04fPb1nsatzK2PDta3vhyLUw\n",
      "https://gmbinder.com/share/-MZUw206A0fXazeekO4G\n",
      "https://homebrewery.naturalcrit.com/share/xeQqs58pKN0y\n",
      "https://homebrewery.naturalcrit.com/share/B1SSgJEQGX\n",
      "https://gmbinder.com/share/-MfIk1CA4Ce2Hah9DepWPDF\n",
      "https://homebrewery.naturalcrit.com/share/_hqiHet220lz\n",
      "https://homebrewery.naturalcrit.com/share/B4nvUv_1BREn\n",
      "https://gmbinder.com/share/-MUi3eG_bONWBTSQMe-_\n",
      "https://gmbinder.com/share/-LdbtzMhuvWxxE2_RDeW\n",
      "https://homebrewery.naturalcrit.com/share/rJ-evtOmNS\n",
      "https://homebrewery.naturalcrit.com/share/aVj374nc4WZH\n",
      "https://homebrewery.naturalcrit.com/share/RBxwItj487a4\n",
      "https://homebrewery.naturalcrit.com/share/Ead2GjxQcWc5\n",
      "https://homebrewery.naturalcrit.com/share/9CfBXAn2VO4b\n",
      "https://gmbinder.com/share/-N7fdBuuzHJemretcT7P\n",
      "https://homebrewery.naturalcrit.com/share/hQsuH5pUN6N2\n",
      "https://homebrewery.naturalcrit.com/share/P_cVG3IK-7cM\n",
      "https://homebrewery.naturalcrit.com/share/miBV3EZ7z1B1\n",
      "https://gmbinder.com/share/-NBsg__gjy9-SoNDvUrf/-NGgq5OZDnJgDjJeecx\n",
      "https://homebrewery.naturalcrit.com/share/h1CgB91Na-F1\n",
      "https://homebrewery.naturalcrit.com/share/khX8q8uq5sgl\n",
      "https://homebrewery.naturalcrit.com/share/SJkLv-WO\n",
      "https://gmbinder.com/share/-MURlZA5NtgCsCX7AD4j\n",
      "https://homebrewery.naturalcrit.com/share/BrIpIByO\n",
      "https://homebrewery.naturalcrit.com/share/vhJs57fV3\n",
      "https://homebrewery.naturalcrit.com/share/6H8YooB-3PBK\n",
      "https://homebrewery.naturalcrit.com/share/rJOLlx_g\n",
      "https://homebrewery.naturalcrit.com/share/xqxpI9d-i\n",
      "https://gmbinder.com/share/-Meq5U92S5C_Nj15C_t-5C_qrF\n",
      "https://homebrewery.naturalcrit.com/share/ry107DXO\n",
      "https://homebrewery.naturalcrit.com/share/aESfqWWJc\n",
      "https://homebrewery.naturalcrit.com/share/HyWNw2pmH\n",
      "https://gmbinder.com/share/-MBoRU3YqBjIc-mOokC8\n",
      "https://homebrewery.naturalcrit.com/new\n",
      "https://homebrewery.naturalcrit.com/share/HJBxKaFV04\n",
      "https://gmbinder.com/share/-Ma4BsG3nADkbz7rCwu8\n",
      "https://homebrewery.naturalcrit.com/share/_LLalJhyQg7k\n",
      "https://homebrewery.naturalcrit.com/share/trfFtuEqaFqG\n",
      "https://homebrewery.naturalcrit.com/share/1R5ruE_FXKoAWX5L6VsecuxdAkOfBxsnlAEmxTvn-0edE\n",
      "https://homebrewery.naturalcrit.com/share/1v0S0UazmQDF3LIFCGrqwoyDd1HS6UMlIN3wnCttzSMN0\n",
      "Dropped 85 instances of unsuccessfully retrieved content\n"
     ]
    }
   ],
   "source": [
    "if has_new_links:\n",
    "    to_drop = [\n",
    "        # Drop GMBinder links that are dead.\n",
    "        new_links['Text'].str.contains('NoSuchKey'),\n",
    "        # Drop Homebrwery links that are dead\n",
    "        new_links['Text'].str.contains('Can not find brew'),\n",
    "        # Some users have relocated their files to other sources and are also being pruned here.\n",
    "        new_links['Text'].str.contains('This content has been moved'),\n",
    "        # Some are not found\n",
    "        new_links['Text'].str.contains('Error: File not found'),\n",
    "        # There are a few examples of nonsensical text, which is generally short, thus I'm doing a check for anything less than 150 characters\n",
    "        new_links['Text'].str.len() <= 200,\n",
    "        # Pernicious watercolor links...\n",
    "        new_links['Text'].str.contains(\"# Full Page Watercolor Stains\"),\n",
    "        # Drop anything totally empty.\n",
    "        new_links['Text'].isna()\n",
    "        ]\n",
    "\n",
    "    drop_mask = None\n",
    "    for mask in to_drop:\n",
    "        if isinstance(drop_mask, pd.Series) or isinstance(drop_mask, np.ndarray):\n",
    "            drop_mask = drop_mask | mask\n",
    "        else:\n",
    "            drop_mask = mask\n",
    "    print(*new_links[drop_mask]['link'].unique(), sep='\\n')\n",
    "    dropped = new_links[drop_mask]\n",
    "    new_links = new_links[~drop_mask]\n",
    "    print(f\"Dropped {sum(drop_mask)} instances of unsuccessfully retrieved content\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b3b447b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_links = new_links.sort_values(['manually_reviewed', \"UID\"], ascending=[False, True])\n",
    "new_links = new_links[~new_links.duplicated(\"src_url\", keep='first')]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "49ff83e4",
   "metadata": {},
   "source": [
    "Clicking through the above links confirms no useful content was collected."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "478b4edd-4591-4b4f-b68d-46b1ec8e1ea4",
   "metadata": {},
   "source": [
    "# Saving collected data\n",
    "\n",
    "Lastly, I'll merge, filter, and save the newly collected data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "741c50c4-2ce9-4704-beb9-57cb0e7812ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UID</th>\n",
       "      <th>link</th>\n",
       "      <th>submission_author</th>\n",
       "      <th>submission_id</th>\n",
       "      <th>submission_title</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>submission_flair</th>\n",
       "      <th>submission_score</th>\n",
       "      <th>submission_upvote_ratio</th>\n",
       "      <th>submission_date</th>\n",
       "      <th>comment_author</th>\n",
       "      <th>comment_id</th>\n",
       "      <th>comment_score</th>\n",
       "      <th>comment_body</th>\n",
       "      <th>comment_date</th>\n",
       "      <th>src_url</th>\n",
       "      <th>manually_reviewed</th>\n",
       "      <th>related_link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1103</th>\n",
       "      <td>1510</td>\n",
       "      <td>https://gmbinder.com/share/-M-kCpCFpxZA3chLFuOF</td>\n",
       "      <td>RSquared</td>\n",
       "      <td>oov1no</td>\n",
       "      <td>Martial Prowess 2.1: Expanding martial combat ...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Compendium</td>\n",
       "      <td>228.0</td>\n",
       "      <td>0.98</td>\n",
       "      <td>2021-07-21 17:46:21</td>\n",
       "      <td>RSquared</td>\n",
       "      <td>h611j1j</td>\n",
       "      <td>6.0</td>\n",
       "      <td>PDF: [https://drive.google.com/file/d/1212E9yQ...</td>\n",
       "      <td>2021-07-21 17:49:17</td>\n",
       "      <td>https://www.gmbinder.com/share/-M-kCpCFpxZA3ch...</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1741</th>\n",
       "      <td>2272</td>\n",
       "      <td>https://gmbinder.com/share/-M-kCpCFpxZA3chLFuO...</td>\n",
       "      <td>RSquared</td>\n",
       "      <td>scfbxq</td>\n",
       "      <td>Martial Prowess v2.3: A 5E Tome of Battle with...</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Compendium</td>\n",
       "      <td>341.0</td>\n",
       "      <td>0.99</td>\n",
       "      <td>NaN</td>\n",
       "      <td>RSquared</td>\n",
       "      <td>hu5oydh</td>\n",
       "      <td>6.0</td>\n",
       "      <td>PDF: [https://drive.google.com/file/d/1p0vEwqH...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.gmbinder.com/share/-M-kCpCFpxZA3ch...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1739</th>\n",
       "      <td>2270</td>\n",
       "      <td>https://gmbinder.com/share/-Muk72f9o48Y_sxYWEs...</td>\n",
       "      <td>Sensitive_Coyote_865</td>\n",
       "      <td>svicxd</td>\n",
       "      <td>Trickster's Quirk - A cantrip</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Spell</td>\n",
       "      <td>1062.0</td>\n",
       "      <td>0.99</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sensitive_Coyote_865</td>\n",
       "      <td>hxlhmxp</td>\n",
       "      <td>1.0</td>\n",
       "      <td>How about this one? https://www.gmbinder.com/s...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.gmbinder.com/share/-Muk72f9o48Y_sx...</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>463</th>\n",
       "      <td>472</td>\n",
       "      <td>https://gmbinder.com/share/-Muk72f9o48Y_sxYWEsD</td>\n",
       "      <td>Sensitive_Coyote_865</td>\n",
       "      <td>shyemc</td>\n",
       "      <td>Summon weapon (corrected) - A cantrip</td>\n",
       "      <td>UnearthedArcana</td>\n",
       "      <td>Spell</td>\n",
       "      <td>513.0</td>\n",
       "      <td>0.97</td>\n",
       "      <td>2022-02-01 16:03:05</td>\n",
       "      <td>Sensitive_Coyote_865</td>\n",
       "      <td>hv5ay6b</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Here is a simple conjuration cantrip I designe...</td>\n",
       "      <td>2022-02-01 16:04:44</td>\n",
       "      <td>https://www.gmbinder.com/share/-Muk72f9o48Y_sx...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2320</th>\n",
       "      <td>2920</td>\n",
       "      <td>https://gmbinder.com/share/-MwdP3QqzF1UxUiy5MD...</td>\n",
       "      <td>St0rmyknight</td>\n",
       "      <td>t0qic7</td>\n",
       "      <td>Artificer Rebrewed Alchemist</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Class</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.81</td>\n",
       "      <td>NaN</td>\n",
       "      <td>St0rmyknight</td>\n",
       "      <td>hyd7b0b</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Here is the new iteration, new to GM Binder an...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.gmbinder.com/share/-MwdP3QqzF1UxUi...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2173</th>\n",
       "      <td>2745</td>\n",
       "      <td>https://gmbinder.com/share/-MwdP3QqzF1UxUiy5MDy</td>\n",
       "      <td>St0rmyknight</td>\n",
       "      <td>t0qic7</td>\n",
       "      <td>Artificer Rebrewed Alchemist</td>\n",
       "      <td>DnDHomebrew</td>\n",
       "      <td>Class</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.81</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.gmbinder.com/share/-MwdP3QqzF1UxUi...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       UID                                               link  \\\n",
       "1103  1510    https://gmbinder.com/share/-M-kCpCFpxZA3chLFuOF   \n",
       "1741  2272  https://gmbinder.com/share/-M-kCpCFpxZA3chLFuO...   \n",
       "1739  2270  https://gmbinder.com/share/-Muk72f9o48Y_sxYWEs...   \n",
       "463    472    https://gmbinder.com/share/-Muk72f9o48Y_sxYWEsD   \n",
       "2320  2920  https://gmbinder.com/share/-MwdP3QqzF1UxUiy5MD...   \n",
       "2173  2745    https://gmbinder.com/share/-MwdP3QqzF1UxUiy5MDy   \n",
       "\n",
       "         submission_author submission_id  \\\n",
       "1103              RSquared        oov1no   \n",
       "1741              RSquared        scfbxq   \n",
       "1739  Sensitive_Coyote_865        svicxd   \n",
       "463   Sensitive_Coyote_865        shyemc   \n",
       "2320          St0rmyknight        t0qic7   \n",
       "2173          St0rmyknight        t0qic7   \n",
       "\n",
       "                                       submission_title        subreddit  \\\n",
       "1103  Martial Prowess 2.1: Expanding martial combat ...  UnearthedArcana   \n",
       "1741  Martial Prowess v2.3: A 5E Tome of Battle with...  UnearthedArcana   \n",
       "1739                      Trickster's Quirk - A cantrip  UnearthedArcana   \n",
       "463               Summon weapon (corrected) - A cantrip  UnearthedArcana   \n",
       "2320                       Artificer Rebrewed Alchemist      DnDHomebrew   \n",
       "2173                       Artificer Rebrewed Alchemist      DnDHomebrew   \n",
       "\n",
       "     submission_flair  submission_score  submission_upvote_ratio  \\\n",
       "1103       Compendium             228.0                     0.98   \n",
       "1741       Compendium             341.0                     0.99   \n",
       "1739            Spell            1062.0                     0.99   \n",
       "463             Spell             513.0                     0.97   \n",
       "2320            Class               3.0                     0.81   \n",
       "2173            Class               3.0                     0.81   \n",
       "\n",
       "          submission_date        comment_author comment_id  comment_score  \\\n",
       "1103  2021-07-21 17:46:21              RSquared    h611j1j            6.0   \n",
       "1741                  NaN              RSquared    hu5oydh            6.0   \n",
       "1739                  NaN  Sensitive_Coyote_865    hxlhmxp            1.0   \n",
       "463   2022-02-01 16:03:05  Sensitive_Coyote_865    hv5ay6b            1.0   \n",
       "2320                  NaN          St0rmyknight    hyd7b0b            1.0   \n",
       "2173                  NaN                   NaN        NaN            NaN   \n",
       "\n",
       "                                           comment_body         comment_date  \\\n",
       "1103  PDF: [https://drive.google.com/file/d/1212E9yQ...  2021-07-21 17:49:17   \n",
       "1741  PDF: [https://drive.google.com/file/d/1p0vEwqH...                  NaN   \n",
       "1739  How about this one? https://www.gmbinder.com/s...                  NaN   \n",
       "463   Here is a simple conjuration cantrip I designe...  2022-02-01 16:04:44   \n",
       "2320  Here is the new iteration, new to GM Binder an...                  NaN   \n",
       "2173                                                NaN                  NaN   \n",
       "\n",
       "                                                src_url manually_reviewed  \\\n",
       "1103  https://www.gmbinder.com/share/-M-kCpCFpxZA3ch...              True   \n",
       "1741  https://www.gmbinder.com/share/-M-kCpCFpxZA3ch...              True   \n",
       "1739  https://www.gmbinder.com/share/-Muk72f9o48Y_sx...              True   \n",
       "463   https://www.gmbinder.com/share/-Muk72f9o48Y_sx...               NaN   \n",
       "2320  https://www.gmbinder.com/share/-MwdP3QqzF1UxUi...             False   \n",
       "2173  https://www.gmbinder.com/share/-MwdP3QqzF1UxUi...               NaN   \n",
       "\n",
       "     related_link  \n",
       "1103          NaN  \n",
       "1741         True  \n",
       "1739        False  \n",
       "463           NaN  \n",
       "2320          NaN  \n",
       "2173         True  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df = pd.concat([collected_df, new_links.drop(columns=['Text','idx'])])\n",
    "final_df['submission_flair'] = final_df['submission_flair'].replace('Feature', 'Feat')\n",
    "final_df = final_df.sort_values(['manually_reviewed', \"UID\"], ascending=[False, True])\n",
    "final_df[final_df.duplicated([\"src_url\", \"submission_flair\"], keep=False)].sort_values(\"src_url\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7d212b41",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UID</th>\n",
       "      <th>link</th>\n",
       "      <th>submission_author</th>\n",
       "      <th>submission_id</th>\n",
       "      <th>submission_title</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>submission_flair</th>\n",
       "      <th>submission_score</th>\n",
       "      <th>submission_upvote_ratio</th>\n",
       "      <th>submission_date</th>\n",
       "      <th>comment_author</th>\n",
       "      <th>comment_id</th>\n",
       "      <th>comment_score</th>\n",
       "      <th>comment_body</th>\n",
       "      <th>comment_date</th>\n",
       "      <th>src_url</th>\n",
       "      <th>manually_reviewed</th>\n",
       "      <th>related_link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [UID, link, submission_author, submission_id, submission_title, subreddit, submission_flair, submission_score, submission_upvote_ratio, submission_date, comment_author, comment_id, comment_score, comment_body, comment_date, src_url, manually_reviewed, related_link]\n",
       "Index: []"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df[final_df['UID'].duplicated(keep=False)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "da480fe4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UID</th>\n",
       "      <th>link</th>\n",
       "      <th>submission_author</th>\n",
       "      <th>submission_id</th>\n",
       "      <th>submission_title</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>submission_flair</th>\n",
       "      <th>submission_score</th>\n",
       "      <th>submission_upvote_ratio</th>\n",
       "      <th>submission_date</th>\n",
       "      <th>comment_author</th>\n",
       "      <th>comment_id</th>\n",
       "      <th>comment_score</th>\n",
       "      <th>comment_body</th>\n",
       "      <th>comment_date</th>\n",
       "      <th>src_url</th>\n",
       "      <th>manually_reviewed</th>\n",
       "      <th>related_link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [UID, link, submission_author, submission_id, submission_title, subreddit, submission_flair, submission_score, submission_upvote_ratio, submission_date, comment_author, comment_id, comment_score, comment_body, comment_date, src_url, manually_reviewed, related_link]\n",
       "Index: []"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df = final_df[~final_df.duplicated([\"src_url\", \"submission_flair\"], keep='first')]\n",
    "final_df[final_df.duplicated(\"src_url\", keep=False)].sort_values(\"src_url\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "552950a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "submissions = final_df['comment_body'].isna() & final_df[\"comment_id\"].isna()\n",
    "final_df.loc[submissions, \"related_link\"] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0ef7c6b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 3250 entries, 2 to 2335\n",
      "Data columns (total 18 columns):\n",
      " #   Column                   Non-Null Count  Dtype  \n",
      "---  ------                   --------------  -----  \n",
      " 0   UID                      3250 non-null   int64  \n",
      " 1   link                     3250 non-null   object \n",
      " 2   submission_author        3237 non-null   object \n",
      " 3   submission_id            3250 non-null   object \n",
      " 4   submission_title         3250 non-null   object \n",
      " 5   subreddit                3250 non-null   object \n",
      " 6   submission_flair         3235 non-null   object \n",
      " 7   submission_score         3250 non-null   float64\n",
      " 8   submission_upvote_ratio  3247 non-null   float64\n",
      " 9   submission_date          2009 non-null   object \n",
      " 10  comment_author           2628 non-null   object \n",
      " 11  comment_id               2628 non-null   object \n",
      " 12  comment_score            2628 non-null   float64\n",
      " 13  comment_body             2628 non-null   object \n",
      " 14  comment_date             1867 non-null   object \n",
      " 15  src_url                  3250 non-null   object \n",
      " 16  manually_reviewed        1425 non-null   object \n",
      " 17  related_link             1472 non-null   object \n",
      "dtypes: float64(3), int64(1), object(14)\n",
      "memory usage: 482.4+ KB\n"
     ]
    }
   ],
   "source": [
    "final_df.to_pickle(DATA/'Metadata.pkl')\n",
    "final_df.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fa8def29-d947-4db2-8754-43862c1361eb",
   "metadata": {},
   "source": [
    "# Export .txt versions of each file.\n",
    "\n",
    "Lastly, I'd like to export a .txt version of each file to have a nicer manual inspection format to work with than the main CSV.\n",
    "\n",
    "This also allows me to share the meta-data and labels of the processed examples without infringing on anyone else's data.\n",
    "\n",
    "The name will be {uid}.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "bd126909-02b6-43dc-933b-6ecb3edd022c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3786 texts already collected. Writing 0 text files.\n"
     ]
    }
   ],
   "source": [
    "raw_uids = data_io.get_uids_from_path(TEXT)\n",
    "to_write = new_links[~new_links['UID'].isin(raw_uids)].copy()\n",
    "print(f\"{len(raw_uids)} texts already collected. Writing {len(to_write)} text files.\")\n",
    "\n",
    "for _, row in to_write.iterrows():\n",
    "    data_io.write_row_to_file(row, TEXT)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "75328c47",
   "metadata": {},
   "source": [
    "## Make clean copies of texts"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "159f409c",
   "metadata": {},
   "source": [
    "**Note**: The cleaning function requires some human help\n",
    "\n",
    "Occasionally, human errors/typos (typically just missing `}` CSS closing brackets) produce errors.\n",
    "\n",
    "I've kept the texts as simple .txt files in part for this reason to make it easier to manually fix these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "73248804",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3786 raw texts found\n",
      "0 clean texts found\n",
      "Limiting cleaning only to files in the provided dataframe\n",
      "Cleaning 3250 texts, starting with UID 1.\n",
      "Note processes ~4-6 texts/sec.\n",
      "ETA: 9 min 1 sec to 13 min 32 sec\n"
     ]
    }
   ],
   "source": [
    "clean_raw_scrapes(final_df, raw_path=TEXT, clean_path=CLEAN_TEXT, limit_to_df=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7a3d121a",
   "metadata": {},
   "source": [
    "## Manually review  unusual flair\n",
    "Since /r/DnDHomebrew has optional flair and often less informative, I'll be checking all of these.\n",
    "Similarly, a few flair are un-common and possibly ineffective even on /r/UnearthedArcana, and I'll review these to see if they might be better categorized as something more meaningful"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "714b29f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "submission_flair\n",
       "5e              46\n",
       "Adventure        4\n",
       "Background       8\n",
       "Class          470\n",
       "Compendium     146\n",
       "Feat           118\n",
       "Item           163\n",
       "Mechanic        96\n",
       "Monster        558\n",
       "Official         1\n",
       "Other           12\n",
       "Prestige         3\n",
       "Race           228\n",
       "Resource        12\n",
       "Spell          200\n",
       "Subclass      1166\n",
       "World            4\n",
       "Name: UID, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df.groupby(by=[\"submission_flair\"])[\"UID\"].agg(\"count\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a3e40ca9",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAJOR_LABELS = [\n",
    "    \"Class\",\n",
    "    \"Item\",\n",
    "    \"Monster\",\n",
    "    \"Race\",\n",
    "    \"Spell\",\n",
    "    \"Subclass\",\n",
    "    \"Feat\",\n",
    "    \"Compendium\",\n",
    "    \"Mechanic\",\n",
    "]\n",
    "good_flair = final_df['submission_flair'].isin(MAJOR_LABELS)\n",
    "manually_reviewed = final_df['manually_reviewed']\n",
    "good_labels = good_flair\n",
    "bad_labels = final_df[~good_labels].copy()\n",
    "bad_labels['manually_reviewed'] = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "7e75a88f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 new labels to review.\n",
      "371/371 labels have been manually reviewed.\n"
     ]
    }
   ],
   "source": [
    "bad_labels_path = PRAW/'LabelsToReview.csv'\n",
    "if bad_labels_path.is_file():\n",
    "    # Read in reviewed labels\n",
    "    reviewed_labels = pd.read_csv(bad_labels_path)\n",
    "    reviewed_labels['manually_reviewed'] = reviewed_labels['manually_reviewed'].fillna(False)\n",
    "    reviewed_labels = reviewed_labels[reviewed_labels['manually_reviewed']]\n",
    "    \n",
    "    # Update with any missing labels\n",
    "    new_to_review = ~bad_labels['src_url'].isin(reviewed_labels['src_url'])\n",
    "    print(f\"{new_to_review.sum()} new labels to review.\")\n",
    "    new_to_review = bad_labels[new_to_review].copy()\n",
    "    bad_labels = pd.concat([reviewed_labels, new_to_review])\n",
    "    print(f\"{bad_labels['manually_reviewed'].sum()}/{len(bad_labels)} labels have been manually reviewed.\")\n",
    "\n",
    "bad_labels.to_csv(bad_labels_path, index=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f47fa7d2",
   "metadata": {},
   "source": [
    "### An example of how the review tool looks\n",
    "I haven't figured out how to use this outside the command line yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "02e07c69",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m\u001b[2m371 records already processed out of 371\u001b[0m\n",
      "\u001b[0m\u001b[0m"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!{sys.executable} ./label_review_tool.py ingest -p 55 -d ./data/praw_data/LabelsToReview.csv -m ./data/Metadata.pkl -t ./data/Text_Files/ --clean"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py310",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}