notebooks/analyze-mb-comment-structure.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyzing the structure of Media Bridge–submitted comments\n",
    "\n",
    "This notebook analyzes the comments uploaded by Media Bridge to FCC Docket 17-108, with a focus on understanding the structure behind the algorithmically-generated ones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load the comments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import re\n",
    "import json\n",
    "import math\n",
    "from functools import reduce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Media Bridge uploaded 1.9 million comments in total:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1856553"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb_comments = (\n",
    "    pd.read_csv(\n",
    "        \"../data/bulk-uploads-17-108-with-uuids.csv\",\n",
    "        usecols = [ \"uploader\", \"comments\", \"email_address\" ],\n",
    "        dtype = str,\n",
    "    )\n",
    "    .loc[lambda df: df[\"uploader\"] == \"shane@mediabridgellc.com\"]\n",
    "    .assign(\n",
    "        comments = lambda df: df[\"comments\"].str.replace(u\"\\xa0\", \" \")\n",
    "    )\n",
    ")\n",
    "\n",
    "len(mb_comments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some, however, are duplicates. There are 1.5 million unique comments, where uniqueness is defined as the combination of the comment text and the email address associated with the comment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1501759"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb_deduped = (\n",
    "    mb_comments\n",
    "    .drop_duplicates(subset = [ \"comments\", \"email_address\" ])\n",
    ")\n",
    "\n",
    "len(mb_deduped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Separate randomized vs. non-randomized comments\n",
    "\n",
    "About 472,000 of the comments have no internal randomization; they come from one of five pre-written variations. (One of those five has two sub-variations that differ only in formattng; as a result, there are six strings listed below.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "non_randomized = [\n",
    "    \"The Title II order created a gaping gap in privacy protections by taking the best cop, the FTC, off the beat. That is reason enough to support Chairman Pai's proposal to restore Internet freedom. Restore privacy by repealing Net Neutrality.\",\n",
    "    \"Title II is a Depression-era regulatory framework designed for a telephone monopoly that no longer exists. It was wrong to apply it to the Internet and the FCC should repeal it and go back to the free-market approach that worked so well.\",\n",
    "    \"The free-market Internet was an incredible engine of economic growth, innovation, and job creation since the 1990s and has already been substantially slowed by the 2015 Net Neutrality rules. The slowdown in investment is destroying jobs and risks a big future tax hike to make up for lost private investment. Save American jobs by repealing Net Neutrality.\",\n",
    "    \"The FCC's Net Neutrality rules were written in the Obama White House by political staff and Tech Industry special interests who overruled the FCC's own experts. The FCC's own chief economist Tim Brennan called the rules \\\"an economics-free zone.\\\" They should be repealed.\",\n",
    "    \"Obama's Net Neutrality order was the corrupt result of a corrupt process controlled by Silicon Valley special interests. It gives some of the biggest companies in the world a free ride at the expense of consumers and should be immediately repealed!\",\n",
    "    ' \"Obama\\'s Net Neutrality order was the corrupt result of a corrupt process controlled by Silicon Valley special interests. It gives some of the biggest companies in the world a free ride at the expense of consumers and should be immediately repealed!\"',\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "471677"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb_deduped_nonrandom = (\n",
    "    mb_deduped\n",
    "    .loc[lambda df: df[\"comments\"].isin(non_randomized)]\n",
    ")\n",
    "\n",
    "len(mb_deduped_nonrandom)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Title II is a Depression-era regulatory framework designed for a telephone monopoly that no longer exists. It was wrong to apply it to the Internet and the FCC should repeal it and go back to the free-market approach that worked so well.</td>\n",
       "      <td>127501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The Title II order created a gaping gap in privacy protections by taking the best cop, the FTC, off the beat. That is reason enough to support Chairman Pai's proposal to restore Internet freedom. Restore privacy by repealing Net Neutrality.</td>\n",
       "      <td>92884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The free-market Internet was an incredible engine of economic growth, innovation, and job creation since the 1990s and has already been substantially slowed by the 2015 Net Neutrality rules. The slowdown in investment is destroying jobs and risks a big future tax hike to make up for lost private investment. Save American jobs by repealing Net Neutrality.</td>\n",
       "      <td>83072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Obama's Net Neutrality order was the corrupt result of a corrupt process controlled by Silicon Valley special interests. It gives some of the biggest companies in the world a free ride at the expense of consumers and should be immediately repealed!</td>\n",
       "      <td>74809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>The FCC's Net Neutrality rules were written in the Obama White House by political staff and Tech Industry special interests who overruled the FCC's own experts. The FCC's own chief economist Tim Brennan called the rules \"an economics-free zone.\" They should be repealed.</td>\n",
       "      <td>62635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>\"Obama's Net Neutrality order was the corrupt result of a corrupt process controlled by Silicon Valley special interests. It gives some of the biggest companies in the world a free ride at the expense of consumers and should be immediately repealed!\"</td>\n",
       "      <td>30776</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     count\n",
       "Title II is a Depression-era regulatory framewo...  127501\n",
       "The Title II order created a gaping gap in priv...   92884\n",
       "The free-market Internet was an incredible engi...   83072\n",
       "Obama's Net Neutrality order was the corrupt re...   74809\n",
       "The FCC's Net Neutrality rules were written in ...   62635\n",
       " \"Obama's Net Neutrality order was the corrupt ...   30776"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    mb_deduped_nonrandom\n",
    "    [\"comments\"]\n",
    "    .value_counts()\n",
    "    .to_frame(\"count\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The remaining 1 million comments are, at least on their surface, unique: No two are exactly the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1030082"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb_deduped_random = (\n",
    "    mb_deduped\n",
    "    .loc[lambda df: ~df[\"comments\"].isin(non_randomized)]\n",
    ")\n",
    "\n",
    "len(mb_deduped_random)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If two or more comments were the same, this cell would throw an error\n",
    "assert mb_deduped_random[\"comments\"].value_counts().max() == 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dear Chairman Pai,  I would like to comment on Internet regulation. I strongly recommend Chairman Pai to repeal Obama's scheme to regulate the web. Americans, as opposed to Washington bureaucrats, should purchase the products they prefer. Obama's scheme to regulate the web is a betrayal of the open Internet. It stopped a free-market system that functioned supremely well for decades with broad bipartisan backing.\n",
      "\n",
      "To the Federal Communications Commission:  I'm concerned about network neutrality regulations. I'd like to request the government to undo The previous administration's order to control the web. Individual citizens, not the FCC, should enjoy whatever products they desire. The previous administration's order to control the web is a exploitation  of net neutrality. It broke a market-based framework that functioned remarkably smoothly for many years with nearly universal backing.\n",
      "\n",
      "Chairman Pai:  My comments re: regulations on the Internet. I'd like to suggest Ajit Pai to rescind Obama's scheme to take over the Internet. Internet users, rather than the FCC, should be free to purchase the products they choose. Obama's scheme to take over the Internet is a corruption of the open Internet. It stopped a free-market system that functioned very, very smoothly for decades with both parties' approval.\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\\n\".join(\n",
    "    mb_deduped_random\n",
    "    [\"comments\"]\n",
    "    .sample(3, random_state = 0)\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reverse-engineer the structure of the randomized comments\n",
    "\n",
    "The following code represents BuzzFeed News' best estimate of how the randomized comments were generated.\n",
    "\n",
    "Each sub-list contains the possible variations, which appear to be selected (with equal weighting) at random. Sub-lists with only one item are \"fixed\"; they don't change from comment to comment.\n",
    "\n",
    "One exception is a repeated phrase at the beginning of the fourth sentence of each comment; it repeats whatever happens to have been randomly selected in a particular part of the second sentence. More details on that below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "segments = [\n",
    "    [\n",
    "        \"To whom it may concern:  \",\n",
    "        \"To the Federal Communications Commission:  \",\n",
    "        \"FCC:  \",\n",
    "        \"To the FCC:  \",\n",
    "        \"Dear Commissioners:  \",\n",
    "        \"Dear Mr. Pai,  \",\n",
    "        \"Dear Chairman Pai,  \",\n",
    "        \"Dear FCC,  \",\n",
    "        \"Mr Pai:  \",\n",
    "        \"FCC commissioners,  \",\n",
    "        \"Chairman Pai:  \",\n",
    "        \"\",\n",
    "    ],\n",
    "\n",
    "    [\n",
    "        \"I'm concerned about\",\n",
    "        \"I am concerned about\",\n",
    "        \"I have concerns about\",\n",
    "        \"I'm very concerned about\",\n",
    "        \"I'd like to share my thoughts on\",\n",
    "        \"Hi, I'd like to comment on\",\n",
    "        \"I would like to comment on\",\n",
    "        \"I want to give my opinion on\",\n",
    "        \"I have thoughts on\",\n",
    "        \"I'm contacting you about\",\n",
    "        \"I'm very worried about\",\n",
    "        \"My comments re:\",\n",
    "        \"In reference to\",\n",
    "        \"I am a voter worried about\",\n",
    "        \"I'm a voter worried about\",\n",
    "        \"Regarding\",\n",
    "        \"With respect to\",\n",
    "        \"In the matter of\",\n",
    "    ],\n",
    "\n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"the FCC's so-called Open Internet order\",\n",
    "        \"Internet regulation and net neutrality\",\n",
    "        \"the Obama takeover of the Internet\",        \n",
    "        \"the FCC regulations on the Internet\",\n",
    "        \"network neutrality regulations\",\n",
    "        \"the FCC's Open Internet order\",\n",
    "        \"the FCC rules on the Internet\",\n",
    "        \"net neutrality and Title II\",\n",
    "        \"Net Neutrality and Title II\",\n",
    "        \"regulations on the Internet\",\n",
    "        \"restoring Internet freedom\",\n",
    "        \"net neutrality regulations\",\n",
    "        \"Title 2 and net neutrality\",\n",
    "        \"the future of the Internet\",\n",
    "        \"the Open Internet order\",\n",
    "        \"internet regulations\",\n",
    "        \"net neutrality rules\",\n",
    "        \"Internet regulation\",\n",
    "        \"Network Neutrality\",\n",
    "        \"an open Internet\",\n",
    "        \"Internet freedom\",\n",
    "        \"Internet Freedom\",\n",
    "        \"Net neutrality\",\n",
    "        \"net neutrality\",\n",
    "        \"NET NEUTRALITY\",\n",
    "        \"Title II rules\",\n",
    "    ],\n",
    "    \n",
    "    [ \". I\" ],\n",
    "\n",
    "    [\n",
    "        \"'d like to\",\n",
    "        \" would like to\",\n",
    "        \" want to\",\n",
    "        \" strongly\",\n",
    "        \"\",\n",
    "    ],\n",
    "    \n",
    "    [\n",
    "        \" \"\n",
    "    ],\n",
    "    \n",
    "    [\n",
    "        \"implore\",\n",
    "        \"ask\",\n",
    "        \"request\",\n",
    "        \"urge\",\n",
    "        \"encourage\",\n",
    "        \"recommend\",\n",
    "        \"suggest\",\n",
    "        \"demand\",\n",
    "        \"advocate\",\n",
    "    ],\n",
    "    \n",
    "    [ \" \" ],\n",
    "\n",
    "    [\n",
    "        \"you\",\n",
    "        \"the FCC\",\n",
    "        \"the Federal Communications Commission\",\n",
    "        \"the commissioners\",\n",
    "        \"the commission\",\n",
    "        \"Chairman Pai\",\n",
    "        \"Ajit Pai\",\n",
    "        \"the government\"\n",
    "    ],\n",
    "    \n",
    "    [ \" to \" ],\n",
    "    \n",
    "    [\n",
    "        \"undo\",\n",
    "        \"reverse\",\n",
    "        \"repeal\",\n",
    "        \"overturn\",\n",
    "        \"rescind\",\n",
    "    ],\n",
    "\n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"The previous administration's\",\n",
    "        \"The Obama/Wheeler\",\n",
    "        \"President Obama's\",\n",
    "        \"Barack Obama's\",\n",
    "        \"Tom Wheeler's\",\n",
    "        \"Obama's\",\n",
    "    ],\n",
    "\n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"decision\",\n",
    "        \"scheme\",\n",
    "        \"policy\",\n",
    "        \"order\",\n",
    "        \"power grab\",\n",
    "        \"plan\",\n",
    "    ],\n",
    "    \n",
    "    [ \" to \" ],\n",
    "    \n",
    "    [\n",
    "        \"regulate\",\n",
    "        \"control\",\n",
    "        \"take over\",\n",
    "    ],\n",
    "\n",
    "    [ \" \" ],\n",
    "\n",
    "    \n",
    "    [\n",
    "        \"broadband\",\n",
    "        \"the web\",\n",
    "        \"Internet access\",\n",
    "        \"the Internet\",\n",
    "    ],\n",
    "    \n",
    "    [ \". \" ],\n",
    "    \n",
    "    [\n",
    "        \"Internet users\",\n",
    "        \"Individual citizens\",\n",
    "        \"People like me\",\n",
    "        \"Citizens\",\n",
    "        \"Individual Americans\",\n",
    "        \"Americans\",\n",
    "        \"Individuals\",\n",
    "    ],\n",
    "    \n",
    "    [ \", \" ],\n",
    "    \n",
    "    [\n",
    "        \"rather than\",\n",
    "        \"as opposed to\",\n",
    "        \"not\",\n",
    "    ],\n",
    "    \n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"Washington bureaucrats\",\n",
    "        \"Washington\",\n",
    "        \"big government\",\n",
    "        \"so-called experts\",\n",
    "        \"unelected bureaucrats\",\n",
    "        \"the FCC Enforcement Bureau\",\n",
    "        \"the FCC\",\n",
    "    ],\n",
    "    \n",
    "    [ \", \" ],\n",
    "    \n",
    "    [\n",
    "        \"should be able to\",\n",
    "        \"should be empowered to\",\n",
    "        \"should be free to\",\n",
    "        \"ought to\",\n",
    "        \"deserve to\",\n",
    "        \"should\",\n",
    "    ],\n",
    "    \n",
    "    [\n",
    "        \" \",\n",
    "    ],\n",
    "    \n",
    "    [\n",
    "        \"use\",\n",
    "        \"enjoy\",\n",
    "        \"purchase\",\n",
    "        \"buy\",\n",
    "        \"select\",\n",
    "    ],\n",
    "    \n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"the\",\n",
    "        \"whichever\",\n",
    "        \"whatever\",\n",
    "        \"which\",\n",
    "    ],\n",
    "    \n",
    "    [ \" \" ],\n",
    "        \n",
    "    [\n",
    "        \"products\",\n",
    "        \"applications\",\n",
    "        \"services\",\n",
    "    ],\n",
    "    \n",
    "    [ \" \" ],\n",
    "\n",
    "    [\n",
    "        \"they\",\n",
    "        \"we\",\n",
    "    ],\n",
    "    \n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"want\",\n",
    "        \"desire\",\n",
    "        \"prefer\",\n",
    "        \"choose\",\n",
    "    ],\n",
    "    \n",
    "    [ \". \" ],\n",
    "    \n",
    "    [\n",
    "        \"The previous administration's\",\n",
    "        \"The Obama/Wheeler\",\n",
    "        \"President Obama's\",\n",
    "        \"Barack Obama's\",\n",
    "        \"Tom Wheeler's\",\n",
    "        \"Obama's\",\n",
    "    ],\n",
    "\n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"decision\",\n",
    "        \"scheme\",\n",
    "        \"policy\",\n",
    "        \"order\",\n",
    "        \"power grab\",\n",
    "        \"plan\",\n",
    "    ],\n",
    "    \n",
    "    [ \" to \" ],\n",
    "    \n",
    "    [\n",
    "        \"regulate\",\n",
    "        \"control\",\n",
    "        \"take over\",\n",
    "    ],\n",
    "    \n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"broadband\",\n",
    "        \"the web\",\n",
    "        \"Internet access\",\n",
    "        \"the Internet\",\n",
    "    ],\n",
    "    \n",
    "    [ \" is a \" ],\n",
    "    \n",
    "    [\n",
    "        \"exploitation \",\n",
    "        \"distortion\",\n",
    "        \"perversion\",\n",
    "        \"corruption\",\n",
    "        \"betrayal\",\n",
    "    ],\n",
    "    \n",
    "    [ \" of \" ],\n",
    "    \n",
    "    [\n",
    "        \"net neutrality\",\n",
    "        \"the open Internet\",\n",
    "    ],\n",
    "    \n",
    "    [ \". It \" ],\n",
    "    \n",
    "    [\n",
    "        \"disrupted\",\n",
    "        \"undid\",\n",
    "        \"reversed\",\n",
    "        \"ended\",\n",
    "        \"broke\",\n",
    "        \"stopped\",\n",
    "    ],\n",
    "    \n",
    "    [ \" a \" ],\n",
    "    \n",
    "    [\n",
    "        \"light-touch\",\n",
    "        \"pro-consumer\",\n",
    "        \"hands-off\",\n",
    "        \"free-market\",\n",
    "        \"market-based\",\n",
    "    ],\n",
    "    \n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"policy\",\n",
    "        \"system\",\n",
    "        \"approach\",\n",
    "        \"framework\",\n",
    "    ],\n",
    "    \n",
    "    [ \" that \" ],\n",
    "    \n",
    "    [\n",
    "        \"functioned\",\n",
    "        \"performed\",\n",
    "        \"worked\",\n",
    "    ],\n",
    "    \n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"supremely\",\n",
    "        \"very, very\",\n",
    "        \"very\",\n",
    "        \"remarkably\",\n",
    "        \"fabulously\",\n",
    "        \"exceptionally\",\n",
    "    ],\n",
    "    \n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"well\",\n",
    "        \"successfully\",\n",
    "        \"smoothly\",\n",
    "    ],\n",
    "    \n",
    "    [ \" for \" ],\n",
    "    \n",
    "    [\n",
    "        \"many years\",\n",
    "        \"decades\",\n",
    "        \"a long time\",\n",
    "        \"two decades\",\n",
    "    ],\n",
    "    \n",
    "    [ \" with \" ],\n",
    "    \n",
    "    [\n",
    "        \"nearly universal\",\n",
    "        \"broad bipartisan\",\n",
    "        \"bipartisan\",\n",
    "        \"both parties'\",\n",
    "        \"Republican and Democrat\",\n",
    "    ],\n",
    "\n",
    "    [ \" \" ],\n",
    "    \n",
    "    [\n",
    "        \"support\",\n",
    "        \"consensus\",\n",
    "        \"approval\",\n",
    "        \"backing\",\n",
    "    ],\n",
    "    \n",
    "    [ \".\" ]\n",
    "    \n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check that pattern fully matches comments\n",
    "\n",
    "Here, we compile the comment segments into a single regular expression, which we use to check whether comments match the reverse-engineered model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def segments_to_pattern(segments):\n",
    "    return re.compile(r\"^\" + r\"\".join(\n",
    "    r\"(\" + r\"|\".join(re.escape(option) for option in seg) + r\")\"\n",
    "        for seg in segments) + r\"$\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "pattern = segments_to_pattern(segments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All comments match (otherwise, the result would be greater than zero):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(re.match(pattern, x) is None for x in mb_deduped_random[\"comments\"].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check that there are no superfluous permutations\n",
    "\n",
    "Although the model above succeeds in matching all comments, so would a model that contained, for example, the entire English language. So here we check whether any individual part of the pattern is superfluous, by incrementally removing each one, and seeing whether the comments still match the pattern. (Here we use a random sample of comments, to speed up the process.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_comments = (\n",
    "    mb_deduped_random\n",
    "    [\"comments\"]\n",
    "    .sample(1000, random_state = 0)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A lack of output for this cell is a good thing;\n",
    "# it means no part of the model is superfluous\n",
    "\n",
    "for i, segment in enumerate(segments):\n",
    "    # For each sub-part of the each segment ...\n",
    "    for j, option in enumerate(segment):\n",
    "        \n",
    "        # Replace the sub-part with \"###\", and then test\n",
    "        # whether the pattern-matching fails. It should fail;\n",
    "        # if it does not, then the sub-part is superfluous.\n",
    "        segments_copy = list([ list(o) for o in segments ])\n",
    "        segments_copy[i][j] = \"###\"\n",
    "        new_pattern = segments_to_pattern(segments_copy)\n",
    "        \n",
    "        num_nonmatching_comments = sum((re.match(new_pattern, x) is None)\n",
    "            for x in sample_comments.values)\n",
    "        \n",
    "        # If all of the comments still match after the \"###\" \n",
    "        # substitution, then the replaced sub-part isn't necessary\n",
    "        # to the model.\n",
    "        if num_nonmatching_comments == 0:\n",
    "            print(i, j, option)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check that segments are randomized independently\n",
    "\n",
    "In some text-generation models, the value of one segment may influence the possible values (or weights for those values) of subsequent segments. Here, we check whether that appears to be true for the actual model that generated these comments.\n",
    "\n",
    "First, we extract the bits of text that each comment has used for each section, skipping the \"fixed\" segments. (Here again we use a random sample of comments, to speed things up.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 4, 6]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FIXED_SEGMENT_INDEX = [ i for i, x in enumerate(segments) if len(x) == 1 ]\n",
    "FIXED_SEGMENT_INDEX[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_permutations(comment):\n",
    "    permutations = [ (i, g) for i, g in enumerate(re.match(pattern, comment).groups())\n",
    "        if i not in FIXED_SEGMENT_INDEX ]\n",
    "    \n",
    "    return pd.DataFrame(\n",
    "        permutations,\n",
    "        columns = [ \"seg_i\", \"option\" ],        \n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example, for the first comment in the sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>seg_i</th>\n",
       "      <th>option</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Dear Chairman Pai,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>I would like to comment on</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>Internet regulation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>strongly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>recommend</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>Chairman Pai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>repeal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>13</td>\n",
       "      <td>Obama's</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "      <td>scheme</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>regulate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>19</td>\n",
       "      <td>the web</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>21</td>\n",
       "      <td>Americans</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>23</td>\n",
       "      <td>as opposed to</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>25</td>\n",
       "      <td>Washington bureaucrats</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>27</td>\n",
       "      <td>should</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>29</td>\n",
       "      <td>purchase</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>31</td>\n",
       "      <td>the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>33</td>\n",
       "      <td>products</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>35</td>\n",
       "      <td>they</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>37</td>\n",
       "      <td>prefer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>39</td>\n",
       "      <td>Obama's</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>41</td>\n",
       "      <td>scheme</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>43</td>\n",
       "      <td>regulate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>45</td>\n",
       "      <td>the web</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>47</td>\n",
       "      <td>betrayal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>49</td>\n",
       "      <td>the open Internet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>51</td>\n",
       "      <td>stopped</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>53</td>\n",
       "      <td>free-market</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>55</td>\n",
       "      <td>system</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>57</td>\n",
       "      <td>functioned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>59</td>\n",
       "      <td>supremely</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31</td>\n",
       "      <td>61</td>\n",
       "      <td>well</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32</td>\n",
       "      <td>63</td>\n",
       "      <td>decades</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33</td>\n",
       "      <td>65</td>\n",
       "      <td>broad bipartisan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>34</td>\n",
       "      <td>67</td>\n",
       "      <td>backing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    seg_i                      option\n",
       "0       0        Dear Chairman Pai,  \n",
       "1       1  I would like to comment on\n",
       "2       3         Internet regulation\n",
       "3       5                    strongly\n",
       "4       7                   recommend\n",
       "5       9                Chairman Pai\n",
       "6      11                      repeal\n",
       "7      13                     Obama's\n",
       "8      15                      scheme\n",
       "9      17                    regulate\n",
       "10     19                     the web\n",
       "11     21                   Americans\n",
       "12     23               as opposed to\n",
       "13     25      Washington bureaucrats\n",
       "14     27                      should\n",
       "15     29                    purchase\n",
       "16     31                         the\n",
       "17     33                    products\n",
       "18     35                        they\n",
       "19     37                      prefer\n",
       "20     39                     Obama's\n",
       "21     41                      scheme\n",
       "22     43                    regulate\n",
       "23     45                     the web\n",
       "24     47                    betrayal\n",
       "25     49           the open Internet\n",
       "26     51                     stopped\n",
       "27     53                 free-market\n",
       "28     55                      system\n",
       "29     57                  functioned\n",
       "30     59                   supremely\n",
       "31     61                        well\n",
       "32     63                     decades\n",
       "33     65            broad bipartisan\n",
       "34     67                     backing"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extract_permutations(sample_comments.iloc[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we create a DataFrame of all extracted segments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>seg_i</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>3</th>\n",
       "      <th>5</th>\n",
       "      <th>7</th>\n",
       "      <th>9</th>\n",
       "      <th>11</th>\n",
       "      <th>13</th>\n",
       "      <th>15</th>\n",
       "      <th>17</th>\n",
       "      <th>...</th>\n",
       "      <th>49</th>\n",
       "      <th>51</th>\n",
       "      <th>53</th>\n",
       "      <th>55</th>\n",
       "      <th>57</th>\n",
       "      <th>59</th>\n",
       "      <th>61</th>\n",
       "      <th>63</th>\n",
       "      <th>65</th>\n",
       "      <th>67</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>comment_i</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Dear Chairman Pai,</td>\n",
       "      <td>I would like to comment on</td>\n",
       "      <td>Internet regulation</td>\n",
       "      <td>strongly</td>\n",
       "      <td>recommend</td>\n",
       "      <td>Chairman Pai</td>\n",
       "      <td>repeal</td>\n",
       "      <td>Obama's</td>\n",
       "      <td>scheme</td>\n",
       "      <td>regulate</td>\n",
       "      <td>...</td>\n",
       "      <td>the open Internet</td>\n",
       "      <td>stopped</td>\n",
       "      <td>free-market</td>\n",
       "      <td>system</td>\n",
       "      <td>functioned</td>\n",
       "      <td>supremely</td>\n",
       "      <td>well</td>\n",
       "      <td>decades</td>\n",
       "      <td>broad bipartisan</td>\n",
       "      <td>backing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>To the Federal Communications Commission:</td>\n",
       "      <td>I'm concerned about</td>\n",
       "      <td>network neutrality regulations</td>\n",
       "      <td>'d like to</td>\n",
       "      <td>request</td>\n",
       "      <td>the government</td>\n",
       "      <td>undo</td>\n",
       "      <td>The previous administration's</td>\n",
       "      <td>order</td>\n",
       "      <td>control</td>\n",
       "      <td>...</td>\n",
       "      <td>net neutrality</td>\n",
       "      <td>broke</td>\n",
       "      <td>market-based</td>\n",
       "      <td>framework</td>\n",
       "      <td>functioned</td>\n",
       "      <td>remarkably</td>\n",
       "      <td>smoothly</td>\n",
       "      <td>many years</td>\n",
       "      <td>nearly universal</td>\n",
       "      <td>backing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Chairman Pai:</td>\n",
       "      <td>My comments re:</td>\n",
       "      <td>regulations on the Internet</td>\n",
       "      <td>'d like to</td>\n",
       "      <td>suggest</td>\n",
       "      <td>Ajit Pai</td>\n",
       "      <td>rescind</td>\n",
       "      <td>Obama's</td>\n",
       "      <td>scheme</td>\n",
       "      <td>take over</td>\n",
       "      <td>...</td>\n",
       "      <td>the open Internet</td>\n",
       "      <td>stopped</td>\n",
       "      <td>free-market</td>\n",
       "      <td>system</td>\n",
       "      <td>functioned</td>\n",
       "      <td>very, very</td>\n",
       "      <td>smoothly</td>\n",
       "      <td>decades</td>\n",
       "      <td>both parties'</td>\n",
       "      <td>approval</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Dear Mr. Pai,</td>\n",
       "      <td>Hi, I'd like to comment on</td>\n",
       "      <td>the FCC rules on the Internet</td>\n",
       "      <td></td>\n",
       "      <td>ask</td>\n",
       "      <td>Ajit Pai</td>\n",
       "      <td>reverse</td>\n",
       "      <td>The Obama/Wheeler</td>\n",
       "      <td>scheme</td>\n",
       "      <td>regulate</td>\n",
       "      <td>...</td>\n",
       "      <td>the open Internet</td>\n",
       "      <td>reversed</td>\n",
       "      <td>hands-off</td>\n",
       "      <td>policy</td>\n",
       "      <td>functioned</td>\n",
       "      <td>remarkably</td>\n",
       "      <td>smoothly</td>\n",
       "      <td>many years</td>\n",
       "      <td>Republican and Democrat</td>\n",
       "      <td>consensus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Mr Pai:</td>\n",
       "      <td>I'm contacting you about</td>\n",
       "      <td>the FCC's Open Internet order</td>\n",
       "      <td></td>\n",
       "      <td>request</td>\n",
       "      <td>the FCC</td>\n",
       "      <td>repeal</td>\n",
       "      <td>The Obama/Wheeler</td>\n",
       "      <td>plan</td>\n",
       "      <td>take over</td>\n",
       "      <td>...</td>\n",
       "      <td>the open Internet</td>\n",
       "      <td>reversed</td>\n",
       "      <td>light-touch</td>\n",
       "      <td>system</td>\n",
       "      <td>performed</td>\n",
       "      <td>very, very</td>\n",
       "      <td>smoothly</td>\n",
       "      <td>many years</td>\n",
       "      <td>Republican and Democrat</td>\n",
       "      <td>backing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "seg_i                                               0   \\\n",
       "comment_i                                                \n",
       "0                                 Dear Chairman Pai,     \n",
       "1          To the Federal Communications Commission:     \n",
       "2                                      Chairman Pai:     \n",
       "3                                      Dear Mr. Pai,     \n",
       "4                                            Mr Pai:     \n",
       "\n",
       "seg_i                              1                               3   \\\n",
       "comment_i                                                               \n",
       "0          I would like to comment on             Internet regulation   \n",
       "1                 I'm concerned about  network neutrality regulations   \n",
       "2                     My comments re:     regulations on the Internet   \n",
       "3          Hi, I'd like to comment on   the FCC rules on the Internet   \n",
       "4            I'm contacting you about   the FCC's Open Internet order   \n",
       "\n",
       "seg_i              5          7               9        11  \\\n",
       "comment_i                                                   \n",
       "0            strongly  recommend    Chairman Pai   repeal   \n",
       "1          'd like to    request  the government     undo   \n",
       "2          'd like to    suggest        Ajit Pai  rescind   \n",
       "3                            ask        Ajit Pai  reverse   \n",
       "4                        request         the FCC   repeal   \n",
       "\n",
       "seg_i                                 13      15         17  ...  \\\n",
       "comment_i                                                    ...   \n",
       "0                                Obama's  scheme   regulate  ...   \n",
       "1          The previous administration's   order    control  ...   \n",
       "2                                Obama's  scheme  take over  ...   \n",
       "3                      The Obama/Wheeler  scheme   regulate  ...   \n",
       "4                      The Obama/Wheeler    plan  take over  ...   \n",
       "\n",
       "seg_i                     49        51            53         55          57  \\\n",
       "comment_i                                                                     \n",
       "0          the open Internet   stopped   free-market     system  functioned   \n",
       "1             net neutrality     broke  market-based  framework  functioned   \n",
       "2          the open Internet   stopped   free-market     system  functioned   \n",
       "3          the open Internet  reversed     hands-off     policy  functioned   \n",
       "4          the open Internet  reversed   light-touch     system   performed   \n",
       "\n",
       "seg_i              59        61          63                       65  \\\n",
       "comment_i                                                              \n",
       "0           supremely      well     decades         broad bipartisan   \n",
       "1          remarkably  smoothly  many years         nearly universal   \n",
       "2          very, very  smoothly     decades            both parties'   \n",
       "3          remarkably  smoothly  many years  Republican and Democrat   \n",
       "4          very, very  smoothly  many years  Republican and Democrat   \n",
       "\n",
       "seg_i             67  \n",
       "comment_i             \n",
       "0            backing  \n",
       "1            backing  \n",
       "2           approval  \n",
       "3          consensus  \n",
       "4            backing  \n",
       "\n",
       "[5 rows x 35 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extracted = (\n",
    "    pd.concat([ extract_permutations(x).assign(comment_i = i)\n",
    "        for i, x in enumerate(sample_comments) ])\n",
    "    .set_index([\n",
    "        \"comment_i\",\n",
    "        \"seg_i\",\n",
    "    ])\n",
    "    [\"option\"]\n",
    "    .unstack()\n",
    ")\n",
    "\n",
    "extracted.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test for the independence of randomization, we calculate the correlation between any two segments in a comment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>seg_a</th>\n",
       "      <th>seg_b</th>\n",
       "      <th>corr</th>\n",
       "      <th>seg_int_a</th>\n",
       "      <th>seg_int_b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>2508</td>\n",
       "      <td>0_</td>\n",
       "      <td>1_Hi, I'd like to comment on</td>\n",
       "      <td>0.035640</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2509</td>\n",
       "      <td>0_Chairman Pai:</td>\n",
       "      <td>1_Hi, I'd like to comment on</td>\n",
       "      <td>-0.043523</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2510</td>\n",
       "      <td>0_Dear Chairman Pai,</td>\n",
       "      <td>1_Hi, I'd like to comment on</td>\n",
       "      <td>-0.030419</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2511</td>\n",
       "      <td>0_Dear Commissioners:</td>\n",
       "      <td>1_Hi, I'd like to comment on</td>\n",
       "      <td>-0.040624</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2512</td>\n",
       "      <td>0_Dear FCC,</td>\n",
       "      <td>1_Hi, I'd like to comment on</td>\n",
       "      <td>-0.005508</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        seg_a                         seg_b      corr  \\\n",
       "2508                       0_  1_Hi, I'd like to comment on  0.035640   \n",
       "2509        0_Chairman Pai:    1_Hi, I'd like to comment on -0.043523   \n",
       "2510   0_Dear Chairman Pai,    1_Hi, I'd like to comment on -0.030419   \n",
       "2511  0_Dear Commissioners:    1_Hi, I'd like to comment on -0.040624   \n",
       "2512            0_Dear FCC,    1_Hi, I'd like to comment on -0.005508   \n",
       "\n",
       "      seg_int_a  seg_int_b  \n",
       "2508          0          1  \n",
       "2509          0          1  \n",
       "2510          0          1  \n",
       "2511          0          1  \n",
       "2512          0          1  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "segment_correlations = (\n",
    "    # Turn each permutation into a dummy variable\n",
    "    extracted\n",
    "    .pipe(pd.get_dummies)\n",
    "    \n",
    "    # Calculate the correlations between them\n",
    "    .corr()\n",
    "    .reset_index()\n",
    "    .rename(columns = { \"index\": \"seg_a\" })\n",
    "    \n",
    "    # Melt the correlation matrix into a long/tidy DataFrame\n",
    "    .melt(\n",
    "        id_vars = [ \"seg_a\" ],\n",
    "        var_name = \"seg_b\",\n",
    "        value_name = \"corr\",\n",
    "    )\n",
    "    .assign(\n",
    "        seg_int_a = lambda df: df[\"seg_a\"].str.extract(r\"^(\\d+)\", expand = False).astype(int),\n",
    "        seg_int_b = lambda df: df[\"seg_b\"].str.extract(r\"^(\\d+)\", expand = False).astype(int),\n",
    "    )\n",
    "    \n",
    "    # Take only the first correlation (A•B instead of both A•B and B•A)\n",
    "    # and ignore self-correlations\n",
    "    .loc[lambda df: df[\"seg_a\"] < df[\"seg_b\"]]\n",
    "    \n",
    "    # Ignore correlations within the same segment, since they are mutually exclusive\n",
    "    .loc[lambda df: df[\"seg_int_a\"] != df[\"seg_int_b\"]]\n",
    ")\n",
    "\n",
    "segment_correlations.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output below demonstrates that are only a handful of pairs with a correlation above 0.15; they are all perfect correlations, meaning that the first segment choice guarantees the second. In this case, whatever is chosen for segments `13-19` is repeated for segments `39-45`. (Segments 14, 16, etc. are all fixed segments, and don't vary at all.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>seg_a</th>\n",
       "      <th>seg_b</th>\n",
       "      <th>corr</th>\n",
       "      <th>seg_int_a</th>\n",
       "      <th>seg_int_b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>29970</td>\n",
       "      <td>13_Barack Obama's</td>\n",
       "      <td>39_Barack Obama's</td>\n",
       "      <td>1.0</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30180</td>\n",
       "      <td>13_Obama's</td>\n",
       "      <td>39_Obama's</td>\n",
       "      <td>1.0</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30390</td>\n",
       "      <td>13_President Obama's</td>\n",
       "      <td>39_President Obama's</td>\n",
       "      <td>1.0</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30600</td>\n",
       "      <td>13_The Obama/Wheeler</td>\n",
       "      <td>39_The Obama/Wheeler</td>\n",
       "      <td>1.0</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30810</td>\n",
       "      <td>13_The previous administration's</td>\n",
       "      <td>39_The previous administration's</td>\n",
       "      <td>1.0</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31020</td>\n",
       "      <td>13_Tom Wheeler's</td>\n",
       "      <td>39_Tom Wheeler's</td>\n",
       "      <td>1.0</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31230</td>\n",
       "      <td>15_decision</td>\n",
       "      <td>41_decision</td>\n",
       "      <td>1.0</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31440</td>\n",
       "      <td>15_order</td>\n",
       "      <td>41_order</td>\n",
       "      <td>1.0</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31650</td>\n",
       "      <td>15_plan</td>\n",
       "      <td>41_plan</td>\n",
       "      <td>1.0</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31860</td>\n",
       "      <td>15_policy</td>\n",
       "      <td>41_policy</td>\n",
       "      <td>1.0</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32070</td>\n",
       "      <td>15_power grab</td>\n",
       "      <td>41_power grab</td>\n",
       "      <td>1.0</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32280</td>\n",
       "      <td>15_scheme</td>\n",
       "      <td>41_scheme</td>\n",
       "      <td>1.0</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32490</td>\n",
       "      <td>17_control</td>\n",
       "      <td>43_control</td>\n",
       "      <td>1.0</td>\n",
       "      <td>17</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32700</td>\n",
       "      <td>17_regulate</td>\n",
       "      <td>43_regulate</td>\n",
       "      <td>1.0</td>\n",
       "      <td>17</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32910</td>\n",
       "      <td>17_take over</td>\n",
       "      <td>43_take over</td>\n",
       "      <td>1.0</td>\n",
       "      <td>17</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33120</td>\n",
       "      <td>19_Internet access</td>\n",
       "      <td>45_Internet access</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33330</td>\n",
       "      <td>19_broadband</td>\n",
       "      <td>45_broadband</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33540</td>\n",
       "      <td>19_the Internet</td>\n",
       "      <td>45_the Internet</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33750</td>\n",
       "      <td>19_the web</td>\n",
       "      <td>45_the web</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  seg_a                             seg_b  \\\n",
       "29970                 13_Barack Obama's                 39_Barack Obama's   \n",
       "30180                        13_Obama's                        39_Obama's   \n",
       "30390              13_President Obama's              39_President Obama's   \n",
       "30600              13_The Obama/Wheeler              39_The Obama/Wheeler   \n",
       "30810  13_The previous administration's  39_The previous administration's   \n",
       "31020                  13_Tom Wheeler's                  39_Tom Wheeler's   \n",
       "31230                       15_decision                       41_decision   \n",
       "31440                          15_order                          41_order   \n",
       "31650                           15_plan                           41_plan   \n",
       "31860                         15_policy                         41_policy   \n",
       "32070                     15_power grab                     41_power grab   \n",
       "32280                         15_scheme                         41_scheme   \n",
       "32490                        17_control                        43_control   \n",
       "32700                       17_regulate                       43_regulate   \n",
       "32910                      17_take over                      43_take over   \n",
       "33120                19_Internet access                45_Internet access   \n",
       "33330                      19_broadband                      45_broadband   \n",
       "33540                   19_the Internet                   45_the Internet   \n",
       "33750                        19_the web                        45_the web   \n",
       "\n",
       "       corr  seg_int_a  seg_int_b  \n",
       "29970   1.0         13         39  \n",
       "30180   1.0         13         39  \n",
       "30390   1.0         13         39  \n",
       "30600   1.0         13         39  \n",
       "30810   1.0         13         39  \n",
       "31020   1.0         13         39  \n",
       "31230   1.0         15         41  \n",
       "31440   1.0         15         41  \n",
       "31650   1.0         15         41  \n",
       "31860   1.0         15         41  \n",
       "32070   1.0         15         41  \n",
       "32280   1.0         15         41  \n",
       "32490   1.0         17         43  \n",
       "32700   1.0         17         43  \n",
       "32910   1.0         17         43  \n",
       "33120   1.0         19         45  \n",
       "33330   1.0         19         45  \n",
       "33540   1.0         19         45  \n",
       "33750   1.0         19         45  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    segment_correlations\n",
    "    .loc[lambda df: df[\"corr\"] > 0.15]\n",
    "    .sort_values(\"seg_a\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output below demonstrates that no segment pairs with a correlation below -0.15, other than the possibilities inherently excluded by the perfect correlations above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>seg_a</th>\n",
       "      <th>seg_b</th>\n",
       "      <th>corr</th>\n",
       "      <th>seg_int_a</th>\n",
       "      <th>seg_int_b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>31015</td>\n",
       "      <td>13_Barack Obama's</td>\n",
       "      <td>39_Tom Wheeler's</td>\n",
       "      <td>-0.203366</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30806</td>\n",
       "      <td>13_Barack Obama's</td>\n",
       "      <td>39_The previous administration's</td>\n",
       "      <td>-0.191720</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30179</td>\n",
       "      <td>13_Barack Obama's</td>\n",
       "      <td>39_Obama's</td>\n",
       "      <td>-0.204086</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30597</td>\n",
       "      <td>13_Barack Obama's</td>\n",
       "      <td>39_The Obama/Wheeler</td>\n",
       "      <td>-0.204086</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30388</td>\n",
       "      <td>13_Barack Obama's</td>\n",
       "      <td>39_President Obama's</td>\n",
       "      <td>-0.200477</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33331</td>\n",
       "      <td>19_the Internet</td>\n",
       "      <td>45_broadband</td>\n",
       "      <td>-0.327781</td>\n",
       "      <td>19</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33749</td>\n",
       "      <td>19_the Internet</td>\n",
       "      <td>45_the web</td>\n",
       "      <td>-0.337228</td>\n",
       "      <td>19</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33123</td>\n",
       "      <td>19_the web</td>\n",
       "      <td>45_Internet access</td>\n",
       "      <td>-0.338160</td>\n",
       "      <td>19</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33332</td>\n",
       "      <td>19_the web</td>\n",
       "      <td>45_broadband</td>\n",
       "      <td>-0.355864</td>\n",
       "      <td>19</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33541</td>\n",
       "      <td>19_the web</td>\n",
       "      <td>45_the Internet</td>\n",
       "      <td>-0.337228</td>\n",
       "      <td>19</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>78 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   seg_a                             seg_b      corr  \\\n",
       "31015  13_Barack Obama's                  39_Tom Wheeler's -0.203366   \n",
       "30806  13_Barack Obama's  39_The previous administration's -0.191720   \n",
       "30179  13_Barack Obama's                        39_Obama's -0.204086   \n",
       "30597  13_Barack Obama's              39_The Obama/Wheeler -0.204086   \n",
       "30388  13_Barack Obama's              39_President Obama's -0.200477   \n",
       "...                  ...                               ...       ...   \n",
       "33331    19_the Internet                      45_broadband -0.327781   \n",
       "33749    19_the Internet                        45_the web -0.337228   \n",
       "33123         19_the web                45_Internet access -0.338160   \n",
       "33332         19_the web                      45_broadband -0.355864   \n",
       "33541         19_the web                   45_the Internet -0.337228   \n",
       "\n",
       "       seg_int_a  seg_int_b  \n",
       "31015         13         39  \n",
       "30806         13         39  \n",
       "30179         13         39  \n",
       "30597         13         39  \n",
       "30388         13         39  \n",
       "...          ...        ...  \n",
       "33331         19         45  \n",
       "33749         19         45  \n",
       "33123         19         45  \n",
       "33332         19         45  \n",
       "33541         19         45  \n",
       "\n",
       "[78 rows x 5 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    segment_correlations\n",
    "    .loc[lambda df: df[\"corr\"] < -0.15]\n",
    "    .sort_values(\"seg_a\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>seg_a</th>\n",
       "      <th>seg_b</th>\n",
       "      <th>corr</th>\n",
       "      <th>seg_int_a</th>\n",
       "      <th>seg_int_b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [seg_a, seg_b, corr, seg_int_a, seg_int_b]\n",
       "Index: []"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    segment_correlations\n",
    "    .loc[lambda df: df[\"corr\"] < -0.15]\n",
    "    .loc[lambda df: ~df[\"seg_int_a\"].isin([ 13, 15, 17, 19 ])]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show the repeated segments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Segments `13-19`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "  [\n",
      "    \"The previous administration's\",\n",
      "    \"The Obama/Wheeler\",\n",
      "    \"President Obama's\",\n",
      "    \"Barack Obama's\",\n",
      "    \"Tom Wheeler's\",\n",
      "    \"Obama's\"\n",
      "  ],\n",
      "  [\n",
      "    \" \"\n",
      "  ],\n",
      "  [\n",
      "    \"decision\",\n",
      "    \"scheme\",\n",
      "    \"policy\",\n",
      "    \"order\",\n",
      "    \"power grab\",\n",
      "    \"plan\"\n",
      "  ],\n",
      "  [\n",
      "    \" to \"\n",
      "  ],\n",
      "  [\n",
      "    \"regulate\",\n",
      "    \"control\",\n",
      "    \"take over\"\n",
      "  ],\n",
      "  [\n",
      "    \" \"\n",
      "  ],\n",
      "  [\n",
      "    \"broadband\",\n",
      "    \"the web\",\n",
      "    \"Internet access\",\n",
      "    \"the Internet\"\n",
      "  ]\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(segments[13:20], indent = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Segments `39-45`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "  [\n",
      "    \"The previous administration's\",\n",
      "    \"The Obama/Wheeler\",\n",
      "    \"President Obama's\",\n",
      "    \"Barack Obama's\",\n",
      "    \"Tom Wheeler's\",\n",
      "    \"Obama's\"\n",
      "  ],\n",
      "  [\n",
      "    \" \"\n",
      "  ],\n",
      "  [\n",
      "    \"decision\",\n",
      "    \"scheme\",\n",
      "    \"policy\",\n",
      "    \"order\",\n",
      "    \"power grab\",\n",
      "    \"plan\"\n",
      "  ],\n",
      "  [\n",
      "    \" to \"\n",
      "  ],\n",
      "  [\n",
      "    \"regulate\",\n",
      "    \"control\",\n",
      "    \"take over\"\n",
      "  ],\n",
      "  [\n",
      "    \" \"\n",
      "  ],\n",
      "  [\n",
      "    \"broadband\",\n",
      "    \"the web\",\n",
      "    \"Internet access\",\n",
      "    \"the Internet\"\n",
      "  ]\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(segments[39:46], indent = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculate possible permutations\n",
    "\n",
    "Below, we calculate the total possible permutations, with care to exclude the perfectly correlated segments (which we do by simply removing them from the calculation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_permutations(segments):\n",
    "    count = reduce(lambda x, y: x * y, map(len, segments))\n",
    "    print(f\"Total permutations: {count:,d}\")\n",
    "    \n",
    "    log = math.log10(count)\n",
    "    print(f\"Log10: {log:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_segments(segments, indices):\n",
    "    return [ s for i, s in enumerate(segments) if i not in indices ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total permutations: 9,584,250,725,597,184,000,000\n",
      "Log10: 21.98\n"
     ]
    }
   ],
   "source": [
    "calculate_permutations(remove_segments(segments, [ 39, 41, 43, 45 ]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "---\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}