01_data_prep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 1. Load Data\n",
    "- 2. Get to know Data\n",
    "- 3. Clean Data\n",
    "    - 3.1 remove weird characters\n",
    "    - 3.2 stopwords\n",
    "    - 3.3 popularity based prefiltering\n",
    "    - 3.4 lemmatization / stemming\n",
    "- 4. Tokenize Data\n",
    "- 5. Save preprocessed data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ---------------------------------------------------------------------- #\n",
    "# 1. Load Data\n",
    "\n",
    "Raw data is gathered from [newsapi](https://newsapi.org/) using [this script](https://gist.github.com/davidlenz/961b5cf501b8f89a8c10cbd45306001b)\n",
    "\n",
    "### Newsapi:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1200\"\n",
       "            height=\"350\"\n",
       "            src=\"https://newsapi.org/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1c27787b9c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import IFrame    \n",
    "display(IFrame(\"https://newsapi.org/\", width=1200, height=350))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use [pandas](https://pandas.pydata.org/) to generally handle the data.\n",
    "\n",
    "### Pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1200\"\n",
       "            height=\"350\"\n",
       "            src=\"https://pandas.pydata.org/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1c27787bd88>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(IFrame(\"https://pandas.pydata.org/\", width=1200, height=350))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('data/sample_5000.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Get to know Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (5000, 4)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Index</th>\n",
       "      <th>date</th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>621031</td>\n",
       "      <td>2017-10-26 15:00:54</td>\n",
       "      <td>eBay launches visual search tools that let you shop using photos from your phone or web0eBay today is launching two new visual search tools that will allow online shoppers to use photos they snap,...</td>\n",
       "      <td>eBay launches visual search tools that let you shop using photos from your phone or web</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>621032</td>\n",
       "      <td>2017-10-26 00:00:00</td>\n",
       "      <td>Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller : The SaltScientists are accusing the seed-and-pesticide giant Monsanto of denying the risks of its latest weedkilling...</td>\n",
       "      <td>Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>621035</td>\n",
       "      <td>2017-10-26 14:41:12</td>\n",
       "      <td>Hard Drive Stats for Q3 2017In Q3 2017, Backblaze introduced both 10 TB and 12 TB hard drives into our data centers, we continued to retire 3 TB and 4 TB hard drives to increase storage density, a...</td>\n",
       "      <td>2017 Hard Drive Reliability by Drive Size and Brand</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>621036</td>\n",
       "      <td>2017-10-26 15:27:11</td>\n",
       "      <td>This is part 1 of the “How Unreal Renders a Frame” series, you can access part 2 and part 3 as well.I was looking around the Unreal source the other day and inspired by some excellent breakdowns o...</td>\n",
       "      <td>How Unreal Renders a Frame</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>621423</td>\n",
       "      <td>2017-10-26 16:02:00</td>\n",
       "      <td>Boeing builds towards its first space taxi testWhen NASA first named Boeing and SpaceX as recipients of the Commercial Crew Program, they originally aimed to end the agency's dependence on Russia ...</td>\n",
       "      <td>Boeing builds towards its first space taxi test</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Index                 date  \\\n",
       "0  621031  2017-10-26 15:00:54   \n",
       "1  621032  2017-10-26 00:00:00   \n",
       "2  621035  2017-10-26 14:41:12   \n",
       "3  621036  2017-10-26 15:27:11   \n",
       "4  621423  2017-10-26 16:02:00   \n",
       "\n",
       "                                                                                                                                                                                                      text  \\\n",
       "0  eBay launches visual search tools that let you shop using photos from your phone or web0eBay today is launching two new visual search tools that will allow online shoppers to use photos they snap,...   \n",
       "1  Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller : The SaltScientists are accusing the seed-and-pesticide giant Monsanto of denying the risks of its latest weedkilling...   \n",
       "2  Hard Drive Stats for Q3 2017In Q3 2017, Backblaze introduced both 10 TB and 12 TB hard drives into our data centers, we continued to retire 3 TB and 4 TB hard drives to increase storage density, a...   \n",
       "3  This is part 1 of the “How Unreal Renders a Frame” series, you can access part 2 and part 3 as well.I was looking around the Unreal source the other day and inspired by some excellent breakdowns o...   \n",
       "4  Boeing builds towards its first space taxi testWhen NASA first named Boeing and SpaceX as recipients of the Commercial Crew Program, they originally aimed to end the agency's dependence on Russia ...   \n",
       "\n",
       "                                                                                     title  \n",
       "0  eBay launches visual search tools that let you shop using photos from your phone or web  \n",
       "1            Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller  \n",
       "2                                      2017 Hard Drive Reliability by Drive Size and Brand  \n",
       "3                                                               How Unreal Renders a Frame  \n",
       "4                                          Boeing builds towards its first space taxi test  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Shape:\", df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Index</th>\n",
       "      <th>date</th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3833</th>\n",
       "      <td>3162454</td>\n",
       "      <td>2018-01-23 13:43:00</td>\n",
       "      <td>Apple's HomePod smart speaker will ship February 9thApple's own vision for the future of home audio, the HomePod smart speaker, will begin shipping on February 9th. Pre-orders for the device open ...</td>\n",
       "      <td>Apple's HomePod smart speaker will ship February 9th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4189</th>\n",
       "      <td>3424385</td>\n",
       "      <td>2018-01-30 13:40:00</td>\n",
       "      <td>Amazon gets into healthcare with Warren Buffet and JPMorganAmazon just took a surprising turn into the healthcare industry, teaming up with investing hero Warren Buffet and New York-based bank JPM...</td>\n",
       "      <td>Amazon gets into healthcare with Warren Buffet and JPMorgan</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Index                 date  \\\n",
       "3833  3162454  2018-01-23 13:43:00   \n",
       "4189  3424385  2018-01-30 13:40:00   \n",
       "\n",
       "                                                                                                                                                                                                         text  \\\n",
       "3833  Apple's HomePod smart speaker will ship February 9thApple's own vision for the future of home audio, the HomePod smart speaker, will begin shipping on February 9th. Pre-orders for the device open ...   \n",
       "4189  Amazon gets into healthcare with Warren Buffet and JPMorganAmazon just took a surprising turn into the healthcare industry, teaming up with investing hero Warren Buffet and New York-based bank JPM...   \n",
       "\n",
       "                                                            title  \n",
       "3833         Apple's HomePod smart speaker will ship February 9th  \n",
       "4189  Amazon gets into healthcare with Warren Buffet and JPMorgan  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option(\"max.colwidth\",200)\n",
    "df.sample(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Datatypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1200\"\n",
       "            height=\"150\"\n",
       "            src=\"https://www.tutorialsteacher.com/python/python-data-types\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1c2613dcfc8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(IFrame(\"https://www.tutorialsteacher.com/python/python-data-types\", width=1200, height=150))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index     int64\n",
       "date     object\n",
       "text     object\n",
       "title    object\n",
       "dtype: object"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data distribution\n",
    "\n",
    "First cast the date column, which is currently of object type, to datetime type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index             int64\n",
       "date     datetime64[ns]\n",
       "text             object\n",
       "title            object\n",
       "dtype: object"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"date\"] = pd.to_datetime(df.date)\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now cast the datetime object to the end/start of a period, for example end of week."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index             int64\n",
      "date     datetime64[ns]\n",
      "text             object\n",
      "title            object\n",
      "week     datetime64[ns]\n",
      "dtype: object\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Index</th>\n",
       "      <th>date</th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "      <th>week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>621031</td>\n",
       "      <td>2017-10-26 15:00:54</td>\n",
       "      <td>eBay launches visual search tools that let you shop using photos from your phone or web0eBay today is launching two new visual search tools that will allow online shoppers to use photos they snap,...</td>\n",
       "      <td>eBay launches visual search tools that let you shop using photos from your phone or web</td>\n",
       "      <td>2017-10-29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>621032</td>\n",
       "      <td>2017-10-26 00:00:00</td>\n",
       "      <td>Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller : The SaltScientists are accusing the seed-and-pesticide giant Monsanto of denying the risks of its latest weedkilling...</td>\n",
       "      <td>Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller</td>\n",
       "      <td>2017-10-29</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Index                date  \\\n",
       "0  621031 2017-10-26 15:00:54   \n",
       "1  621032 2017-10-26 00:00:00   \n",
       "\n",
       "                                                                                                                                                                                                      text  \\\n",
       "0  eBay launches visual search tools that let you shop using photos from your phone or web0eBay today is launching two new visual search tools that will allow online shoppers to use photos they snap,...   \n",
       "1  Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller : The SaltScientists are accusing the seed-and-pesticide giant Monsanto of denying the risks of its latest weedkilling...   \n",
       "\n",
       "                                                                                     title  \\\n",
       "0  eBay launches visual search tools that let you shop using photos from your phone or web   \n",
       "1            Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller   \n",
       "\n",
       "        week  \n",
       "0 2017-10-29  \n",
       "1 2017-10-29  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"week\"] = df.date.dt.to_period(\"W\").dt.to_timestamp('W')\n",
    "print(df.dtypes)\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1c2613dd708>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEiCAYAAADwEwVaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxcZb348c83e7M2aZLubdJSurdQ0hRkK6uAKIhwBcVbBC7KRUX9oYByRVQQRAXvVUQEAQGXspfdAmWTpXSjbdKW7kvSZpm22ZNJZp7fH+ecdJpmmcyS2b7v1yuvzJw558wzM8n5zrN8n0eMMSillEo8SZEugFJKqcjQAKCUUglKA4BSSiUoDQBKKZWgNAAopVSC0gCglFIJSgOACpiIPCIiv4jQc4uIPCwiB0RkeZieY4KINItI8gD7LRSRPeEoQzwQkbdE5OpIl0MdSQNAHBGRHSJSIyJZPtuuFpG3IliscDkJOAsYZ4wpD8UJ7ffvTOe+MWaXMSbbGOMJxfmjhYik2IGt3GfbV0XE9LJtY2RKqYaCBoD4kwJcH+lCDNZA37J7MRHYYYxpCcFzpwR7jmjW8/UZY7qAD4BTfTafAmzsZds7YS+gihgNAPHnbuAGERne8wERKbG/5aX4bOuunovIFSLybxG5R0QOisg2EfmMvX23iNSKyKIepy0UkaUi0iQib4vIRJ9zT7Mf2y8im0TkP3wee0RE/igiL4tIC3BaL+UdIyJL7OO3iMh/2duvAh4ETrC/yd7Wy7GTReRNEXGJSL2IPOH7ntjf9m8UkbVAi4j8HZgAvGCf84c93y8RKbCbnartpqfnevsA7HI/LSJ1IrJdRL7j81i5iKwQkUa7tvbbPs6xUET2iMiP7PLvEJGv+jyeLiK/FpFd9nnuF5FhPY69UUT2AQ/38hTvYF3gHScDd/Wy7R37nEkicpOIbLXf08UiUuBTnuNF5H377+YTEVnYx+saLSJrReSG3h5XQ8wYoz9x8gPsAM4EngF+YW+7GnjLvl0CGCDF55i3gKvt21cAXcDXgWTgF8Au4A9AOnA20ARk2/s/Yt8/xX78d8B79mNZwG77XCnAPKAemOlzbANwItYXkYxeXs/bwH1ABnAMUAec4VPW9/p5L47CaiJKB4qwLmT39niv1gDjgWG+75/PPoe9X8BLwD+BfCAVONXevhDYY99OAlYCPwHSgEnANuCz9uMfAF+zb2cDx/dR/oX2Z/Fb+zWcCrQAU+3H7wWWAAVADvAC8Msex95lHzusl/OfCuy3y1sI7AQygRqfbV5ggr3/d4EPgXH2Of8E/N1+bCzgAs6zjz3Lvl/k+zdmv5+fAtdE+n9Ff+y/g0gXQH9C+GEeCgCz7ItrEYMPAJt9Hptt7z/SZ5sLOMa+/QjwD5/HsgEP1kX1y8C7Pcr3J+BWn2P/2s9rGW+fK8dn2y+BR3zK2mcA6OV8FwKre7xXV/b2/vnc736/gNH2BTG/l3Mv5FAAWADs6vH4zcDD9u13gNuAwgHKuxDrIp7ls20x8D+AYAWDyT6PnQBs9znWTS9B1Wf/DKAdmAt8EXjC3v6hz7btPvtvwA6+9v3RQKf93twIPNbj/K8Bi3z+xn5rv7+XRfr/RH8O/cR122eiMsasF5EXgZuw/nEHo8bndpt9vp7bsn3u7/Z53mYR2Q+MwWqjXyAiB332TQEe6+3YXowB9htjmny27QTK/HkRIlIM/C9WM0YO1jfTAz126+/5expvl6fnOXqaCIzp8bqTgXft21cBPwM2ish24DZjzIt9nOuAObyPYyfW+1KE9W19pYg4j4n9PI46Y0x7X4U0xrSLNXrqFKxailO+93y2+bb/TwSeFRGvzzYPMNJ+7BIR+bzPY6nAMp/7XwW2AE/1VSY19LQPIH7dCvwXVvXc4VxMMn22jQryecY7N0QkG6tJohrr4vq2MWa4z0+2MeZan2P7m4q2GigQkRyfbROAKj/L9Uv7/HOMMbnA5VgXSV89n7+/8uy2y3NE30ov+23v8bpzjDHnARhjNhtjLgOKsZponvIdtdVDfo/HJmC9L/VYgXimz3PkGWN8A7M/0/w6/QAncygAvOuzzTcA7AbO7fG6MowxVfZjj/V4LMsYc6fP8T+1y/23ADr8VZhoAIhTxpgtWO3V3/HZVod1Ab1cRJJF5EpgcpBPdZ6InCQiacDPgY+MMbuBF4GjReRrIpJq/8wXkel+ln838D7wSxHJEJE5WN+en/CzXDlAM3BQRMYCP/DjmBqsb769lWcv8Apwn4jk26/nlF52XQ402h2ww+z3eZaIzAcQkctFpMgY4wWcWkJ/w0xvE5E0ETkZOB940j72z8A9dk0HERkrIp/14zX6eger8308UGlvew+rCekYDg8A9wO3O538IlIkIhfYjz0OfF5EPmu/3gy7I3qcz/GdwCVYfUOPiYhee6KAfgjx7WdY/3C+/gvrYugCZmJdZIPxN6zaxn7gOKyqPnbTzdnApVjfWvdxqFPSX5dhtcNXA89i9R8s9fPY27A6nhuwOm+f8eOYXwK32CNZehul8jWsC9lGoBarY/QwxsoZ+DzWBXQ71rfeB4E8e5dzgAoRacbqNL+0n6aafVjNVtVYge+bxhhnXP6NWE0qH4pII/A6MNWP1+jrfbtcHxljNdYbY1xYne21xpjNPvv+DqvT+V8i0oTVV7DAPmY3cAHwI/vY3Vh/Y4ddX4wxbuAirNrPXzQIRJ7Yn7tSKorYwygfN8aMG2hfpQKlEVgppRKUBgCllEpQ2gSklFIJSmsASimVoDQAKKVUgoqKTODCwkJTUlIS6WIopVRMWblyZb0xpijQ46MiAJSUlLBixYpIF0MppWKKiOwM5nhtAlJKqQSlAUAppRKUBgCllEpQGgCUUipBaQBQSqkEpQFAKaUSlAYAFZWMMXR5vAPvqJQKmAYAFZWe+GgXJ/9qGV6vzlWlVLhoAFBRaeO+RvY2tHOg1R3poigVtzQAqKjkarYu/LVNHREuiVLxSwOAikpOAKhp7Gu1RKVUsDQAqKhU32x989cagFLhowFARSUnANRpAFAqbDQAqKjj7vLS2N4FQK02ASkVNhoAVNTZ33Jo5I82ASkVPhoAVNRxmn9AA4BS4aQBQEUdJwCMyx9GbZM2ASkVLhoAVNRxhoDOGJ1LbWMHxmg2sFLhoAFARR1Xi1UDmD46l44uL41tXREukVLxSQOAijquZjfpKUlMKsoC0GYgpcJEA4CKOnXNHRRmp1OUkw5oR7BS4aIBQEUdV7ObEdlpjMzNALQGoFS4aABQUcfV0sGIrDSKnRpAo9YAlAoHDQAq6ria3RRmp5OdnsKw1GRtAlIqTDQAqKhijLGbgNIREYpz0zUAKBUmGgBUVGls78Lt8VKYnQZAcU66zgekVJhoAFBRxWVnAY/oDgAZWgNQKkw0AKio4rInghuRZXUAF2kNQKmw0QCgoopTAyjMtgJAcW46LW4PLR2aDaxUqGkAUFGl3p4HyOkDGJnj5AJoM5BSoeZXABCR74lIhYisF5G/i0iGiBSIyFIR2Wz/zvfZ/2YR2SIim0Tks+Ervoo3zkyg+Vl2H0CukwugzUBKhdqAAUBExgLfAcqMMbOAZOBS4CbgDWPMFOAN+z4iMsN+fCZwDnCfiCSHp/gq3ria3QzPTCU12frTLNYagFJh428TUAowTERSgEygGrgAeNR+/FHgQvv2BcA/jDEdxpjtwBagPHRFVvHMyQJ2FOt8QEqFzYABwBhTBfwa2AXsBRqMMf8CRhpj9tr77AWK7UPGArt9TrHH3nYYEblGRFaIyIq6urrgXoWKG/V2FrBjeGYqaclJOh+QUmHgTxNQPta3+lJgDJAlIpf3d0gv245Y0cMY84AxpswYU1ZUVORveVWcc9kzgTpExB4KqjUApULNnyagM4Htxpg6Y0wn8AzwGaBGREYD2L9r7f33AON9jh+H1WSk1IDq7ZlAfRXlpGsNQKkw8CcA7AKOF5FMERHgDGADsARYZO+zCHjevr0EuFRE0kWkFJgCLA9tsVU8cnd5aWjr7E4Cc4zM1RqAUuGQMtAOxpiPROQpYBXQBawGHgCygcUichVWkLjE3r9CRBYDlfb+1xljPGEqv4ojB1rtHICcw2sAxTkZfLhtfySKpFRcGzAAABhjbgVu7bG5A6s20Nv+twO3B1c0lWicHICeNYDinHQa2jpp7/SQkaojipUKFc0EVlHD1SML2OEkg9XpUFClQkoDgIoa3TWA7J41AE0GUyocNACoqOHUAHobBQRQpyOBlAopDQAqatS3dJCWkkRO+uFdU04TUI2OBFIqpDQAqKjhanZTmJWGNdr4kBFZ6SQJmgugVIhpAFBRw9XccUT7P0BykmYDKxUOGgBU1OgtC9ihS0MqFXoaAFTUcDV3HJED4CjOSdcAoFSIaQBQUcEYQ32L+4gsYEdxbrqOAlIqxDQAqKjQ3NGFu8tLYR81gKKcDFwtbro83iEumVLxSwOAigr1feQAOIpz0jHm0H5KqeBpAFBRwdVHFrDDWRmsRtcGVipkNACoqNBdA8jqqw9Ap4NQKtQ0AKio4GqxLuzOtA89jcx11gbWGoBSoaIBQEUFZx6g/MzeawCF2emIoMlgSoWQBgAVFeqbO8gblkpaSu9/kqnJSRRkpmkTkFIhpAFARQVXP1nAjqIczQVQKpQ0AKioUN/c0WcOgKM4V6eDUCqUNACoqODqJwvYUawTwikVUhoAVFTobx4gR3FOOnXNHXi8ZohKpVR80wCgIq7T4+VAa+eAfQDFOel4vIb9LZoNrFQoaABQEXegxZkGov8awMjuZDDtCFYqFDQAqIhzsoAL+8gCdhR3J4NpP4BSoaABQEWckwVc2EcWsKM4x6oB1GlHsFIhoQFARZxrgHmAHM40EdoEpFRoaABQEVc/wEygjozUZHIzUrQJSKkQ0QCgIq6+2U1achK5GSkD7lucm6G5AEqFiAYAFXGu5g5GZKchIgPuW5yTTo02ASkVEhoAVMS5WgaeB8ih2cBKhY4GABVx/mQBO0bmZlDX1IExmg2sVLA0AKiIq/djJlBHUU46bo+XhrbOMJdKqfinAUBFlDHGmgl0gBFADl0aUqnQ0QCgIqrF7aGjy0vhIPoAQFcGUyoUNACoiHI5OQB+9gEUazKYUiHjVwAQkeEi8pSIbBSRDSJygogUiMhSEdls/8732f9mEdkiIptE5LPhK76Kdc48QH6PAtImIKVCxt8awO+AV40x04C5wAbgJuANY8wU4A37PiIyA7gUmAmcA9wnIsmhLriKD04WsL99ANnpKWSmJVPTqDUApYI1YAAQkVzgFOAhAGOM2xhzELgAeNTe7VHgQvv2BcA/jDEdxpjtwBagPNQFV/HBNcgaAFhDQbUGoFTw/KkBTALqgIdFZLWIPCgiWcBIY8xeAPt3sb3/WGC3z/F77G2HEZFrRGSFiKyoq6sL6kWo2DXYPgCwF4fXTmClguZPAEgB5gF/NMYcC7RgN/f0obd8/iOydowxDxhjyowxZUVFRX4VVsUfV4ub3IwU0lL8H49QnJOuncBKhYA//3V7gD3GmI/s+09hBYQaERkNYP+u9dl/vM/x44Dq0BRXxZvB5AA4inO0CUipUBgwABhj9gG7RWSqvekMoBJYAiyyty0CnrdvLwEuFZF0ESkFpgDLQ1pqFTfq7YngBqM4N51Wt4fmjq4wlUqpxDDw/LuWbwNPiEgasA34OlbwWCwiVwG7gEsAjDEVIrIYK0h0AdcZYzwhL7mKC65mN5OLsgd1zKFksHayB3msUuoQvwKAMWYNUNbLQ2f0sf/twO1BlEslCFeLmwWTBlkDyDmUCzBJA4BSAdNMYBUxXR4vB1rdgxoBBIcWh9dcAKWCowFARcz+VjfG4Pc8QI6RzuLw2hGsVFA0AKiIOZQENrgaQO4wa9iojgRSKjgaAFTEdAeArMHVAETEXhlMm4CUCoYGABUxrhZ7HqCcwdUAwEkG0xqAUsHQAKAixpkJtHCQncCgyWBKhYIGABUx9c0dpCQJucP8TUc5pDhXm4CUCpYGABUxLjsLWKS36aP6V5yTTmN7F+2dmmOoVKA0AKiIcTUPPgfA0Z0MprOCKhUwDQAqYupb3AF1AMOhZDCdFVSpwGkAUBHjau6gcJBDQB2+00EopQKjAUBFhDEmoJlAHd01AO0IVipgGgBURLS6PbR3egedBewoyEwjJUm0BqBUEDQAqIhwsoAHuxiMIylJKMzWZDClgqEBQEVEvZ0FHGgTENi5ABoAlAqYBgAVEa4gsoAdOh+QUsHRAKAior45+BpAkU4HoVRQNACoiHDZAaAgwGGgACNz09nf4sbd5Q1VsZRKKBoAVETUN7vJyUghIzU54HM4uQBObUIpNTgaAFREuFrcAY8AcnQvDq/NQEoFRAOAighXc8egF4LpSZPBlAqOBgAVEcFkATt0OgilgqMBQEWEq9kdcBawozA7DRENAEoFSgOAGnIer2F/a/B9ACnJSYzISqNOZwRVKiAaANSQO9DqxhjrG3ywinMyqNE1AZQKiAYANeScLOBAF4PxZU0HoTUApQKhAUANuVBkATus6SC0BqBUIDQAqCHnBIBQNQHVN3fg8Zqgz6VUotEAoIZcsFNB+yrOTcdrwNWitQClBksDgBpyrpYOUpKE3IzUoM/VnQ2szUBKDZoGADXkXM1uCrLSSEqSoM9VZCeD1WkugFKDpgFADTkrCzj45h/wnQ9IRwIpNVgaANSQq292h6QDGA7NB6S5AEoNngYANeRcLR0h6QAGSE9JZnhmqtYAlAqA3wFARJJFZLWIvGjfLxCRpSKy2f6d77PvzSKyRUQ2ichnw1FwFbtcze6gZwL1pbkASgVmMDWA64ENPvdvAt4wxkwB3rDvIyIzgEuBmcA5wH0iEviqHyqutLq7aHV7QtYHAFYugE4Ip9Tg+RUARGQc8DngQZ/NFwCP2rcfBS702f4PY0yHMWY7sAUoD01xVazrngYiRH0AYNUAdBSQUoPnbw3gXuCHgO/iqyONMXsB7N/F9vaxwG6f/fbY25QKaRawoyjXCgDGaDawUoMxYAAQkfOBWmPMSj/P2dvg7iP+M0XkGhFZISIr6urq/Dy1inWhzAJ2FOdk4PZ4OdjaGbJzKpUI/KkBnAh8QUR2AP8ATheRx4EaERkNYP+utfffA4z3OX4cUN3zpMaYB4wxZcaYsqKioiBegoolzpQNoe0D0LWBlQrEgAHAGHOzMWacMaYEq3P3TWPM5cASYJG92yLgefv2EuBSEUkXkVJgCrA85CVXMam+eyro0DUBjcy1soFrdG1gpQYlJYhj7wQWi8hVwC7gEgBjTIWILAYqgS7gOmOMJ+iSqrhQ39xBdnoKGamhGximNQClAjOoAGCMeQt4y77tAs7oY7/bgduDLJuKQnVNHRTlBN584wphFrDDyQbWZDClBkczgZXfPtjqovyO11m580DA53C1hG4eIEdmWgrZ6SmaDKbUIGkAUH77eMd+jIEnV+weeOc+hDoL2KG5AInBGKPDfUNIA4DyW0V1AwAvrd1Le2dg3TqhnAnUV1GOrg2cCG57oZLz/+89DQIhogFA+W19VSNjhw+jqaOLpZU1gz7e4zXsbwl9HwBAcW5g00F0ebxc98Qqzvrt2/x26ad8WtMU8rKp0Fi2sZZH3t9BRXUjew60Rbo4cUEDgPLLgRY3VQfbuPz4iYzJy+CZVXsGfY6DrW68JrRJYA5nQrjBfDM0xnDrkgpeWreXYWnJ/P7NzZx9zzucfc/b/O71zWytaw55OVVgDra6ufHptd0jvpZv3x/hEsUHDQDKL5V7GwGYMy6PC48dyzub6wfd5OJqCf08QI6Ruem0dXpo6ujy+5j7397GEx/t4r8XTmbJt07iwx+dwc8umMnwzDTufeNTzvjN25xz7zv8YdkWdtS3hLzMyn8/eb6C/S1uHlo0n7xhqRoAQiSYPACVQNZXWe3/M8fkMjI3nfve2sqSNdVcffIkv8/hzAM0IiscNQArGay2scOvtYafX1PFXa9u5Atzx3DD2VO7z/GfJ5TwnyeUsK+hnVfW7+XFtXu5+7VN3P3aJmaNzeVzs8dw/pzRjC/IDPlrUL17ae1elnxSzffPOprZ4/KYX5LP8h0aAEJBA4Dyy/pqq/1/eGYawzPTmDsuj6dXVQ0yADjzAIVnFBBYuQBHFWf3u++H21z84Mm1LCgt4O5L5vS6NvGovAy+fmIpXz+xlOqDbby8zgoGd726kbte3cjccXmcO3s0x44fzowxueSEYIF7daTapnZueW4dc8fl8d8LJwNQXlrA6xtqqW1sp9jOAleB0QCg/FJR1cCssbnd9y+aN45bl1RQWd3IjDG5/Rx5iKs59PMAOZxksIGGgm6pbeKav65gwohMHvhaGekpA2ckjxk+jKtPnsTVJ09i9/7W7mBw5ysbu/cpGZHJzLF5zByTy8wxecwakxuW15lIjDHc/PQ6Wt0efvMfx5CSbLVYl5eOAGD5jv2cP2dMJIsY8zQAqAE1d3Sx3dXCF489NKv35+eO4ecvVvLs6j3MGDPDr/O4mt0kJwnDh4X+23KRTxNQX2qb2ln0l49JS0nm4Svmk5c5+HKML8jkG6dO5hunTqa2sZ2K6kbWVzVQUd3I2j0HeWnt3u59R+VmMGtsLjPsgDBzbB5j8jIQ6W3CXNXTkyv38MbGWm753PTDanUzx+SSmZbM8u0aAIKlAUANaMPeRoyBmT41gIKsNE6bVsxza6q58Zxp3d/O+uNq6aAgK63XJpdg5WakkJ6S1GfHdEtHF1c+8jH7W9ws/sYJIWnDL87NoDg3g9OmFXdva2jtpGJvAxVVjVRUW4HhzY21eO3BSfmZqdx2wSy+MFcvXP3Zc6CVn71QyYLSAq48sfSwx1KTkzhuYr52BIeABgA1IKcDeNaYvMO2f2neWJZW1vDelnoWTi3u7dDD1IcpCxhARCjOTe81F6DL4+Xbf19NZXUjDy4qY/a4vF7OEBp5mal8ZnIhn5lc2L2tze1hw75GKqobuW/ZFp5auUcDQD+8XsMPnlyLMYZfXzK31y8M5SUF/Pb1TznY6mZ4Znj+phKBDgNVA6qobqQoJ/2IDrfTphUzPDOVZ1ZV+XWe+uaOsOQAOEbmZBzRBOSM9X9zYy0/v3AWp08bGbbn78uwtGTmTcjna8dP5MzpI1m18wAer2ay9uXRD3bwwTYX/3P+jD5rauWlBRgDK3YEPi+V0gCg/LC+qoGZvXT0pqck8/k5Y3itYh9N7QOvxuVqdoclB8BRnJtOTY8mIGes/7ULJ/PVBRPD9tz+KivJp7mjiw12XoU63Na6Zu58ZSMLpxbx5fnj+9xv7vjhpCUn6XDQIGkAUP1q7/Swubb5iOYfx0XzxtLR5eWVdfsGPJcrzDWA4pwM6nxqAL5j/X9gj/WPtPklBQCs0AvXEbo8Xv7f4k/ISE3mri/N6bezPCM1mbnj8/hI+wGCogFA9WvTviY8XnPYEFBfx4wfzqTCLJ4eYGqINreHFrcnrDWAopx0mjq6aHN7/BrrHwljhg9j7PBhfBzElNrx6k/vbGPN7oP8/MJZ3au89ae8tID1VQ20DCL7Oxo1tndy+YMf8c+Pdw35c2sAUP2qqLaaKmb2UQMQES6aN5aPtu9n9/7WPs/jrAVcGIYsYIeTDPb+1vpBj/UfSmUl+azYsV9ntPRRWd3Iva9/yudmj+bzc0b7dUx56Qg8XsOqXbEdTG99voL3ttRz0zPreHnd3oEPCCENAKpf66sbyM1IYVz+sD73udDOD3h2dd+dwd1rAYe1D8D61njd31YFNdY/3MpKCqhp7NAZLW0dXR6+v3gNecPS+PmFs/zOkzhuYj5JEtsTwy35pJpnV1dx7cLJzJuQz3f/sYYPtrqG7Pk1AKh+WRnAef3+U47Lz+T4SQU8s2pPn99qw5kF7HBqAILwlyvKona+nvkl+YC1wI6C372+mY37mrjrS7MpGMQw4ez0FGaNzYvZAFB1sI0fP7uOeROG8//OOpqHFpUxYUQm1/x1xZANEtAAoPrU6fGyYV9TryOAerpo3jh2uFpZtetgr4+7wjgPkKO0MIuTpxRy3+XzmDNueNieJ1hHF+eQk5HCxzqEkZU7D3D/21v5j7JxnDF98EN0y0sKWL37IB1dgS1QFCker+H7/1yD12u458vWNBfDM9P465XlZKWnsOgvy9lzoO8m1VDRAKD6tLWuGXeXl1ljB06cOnfWKDJSk/pcJ6C+JXwzgToyUpN57KoFnOZHUlokJSUJZRPzE34kUKu7ixue/ITRecP4n/P9m06kp/LSAtxdXtbuaQhx6cLrz+9u46Pt+7n1CzOZOCKre/uY4cP461XltHd6+M+/LGe/PYV6uGgAUH1aX9V/B7CvnIxUzpk5ihc+qe7125ir2U1WWjLD0qKrQzZSykoK2FzbzIEw/4NHs7te2cj2+hbuvmROwLOpOsNqY6kZaH1VA7/51ybOnTWKS44bd8TjR4/M4cFF89lzoI0rH/mYVnf4RjlpAFB9Wl/VQGZaMqWFWQPvjNUM1NjexZsbao94LFxrAccq58K1MkGHg67YsZ9HP9jJFZ8pOWzajMHKz0pj6sicmMkHaHN7uP4fqynISuOOL87us2+tvLSA/7vsWNbuOci3/raaTo83LOXRAKD6VFndyPTRuST7OYb+xKMKKc5J5+lepoYIdxZwrJkzLo+05CQ+3hkbF65Qcnd5ufmZdYwdPowffDb4BL35pfms3LGfrjBdJEPpjpc3sLWuhd9ccgz5A3R4f3bmKH52wSze3FjLj55ZF5ZhwxoAVK+8XkNFdQOz/JzrHyA5SfjisWN5a1Nt96gfR7jnAYo1GanJzB6Xl5Bz2TzwzlY21zbzswtmkpUe/HyU5aUjaHF7upctjVZvbqzhsQ93ctVJpZw0xb9az+XHT+Q7Z0zhyZV7+PW/NoW8TBoAVK92uFpocXuY6UcHsK+L5o2jy2tY8kn1YdtdLe6wjgCKRWUl+azdc5D2ztgawRKMHfUt/O+bWzh31qiARv30pjwG+gHqmzv44VNrmTYqZ9C1nu+dOYXLysfzh2VbefT9HSEtlwYA1atDGcD+1wu3cmEAABvvSURBVAAApo7KYeaY3MNmCPV6Dftb3GEdARSL5k8soNNjYm4ES6CMMfz4uXWkJyfx0y/MDNl5R+VlMHFEZtQGAGMMNz61lsb2Lu699BgyUgc3EEJE+PkFszhz+kh++kLFYYsOBUsDgOrV+uoG0pKTmFKcM+hjL5o3jnVVDXxa0wTAwbZOPF6jfQA9HDcxsRLCnltTxb+3uPjhOVP9mutnMMpLCvh4x368UTjN9hMf7eKNjbXceM40po0a3BcqR0pyEr//yrHMm5DP9/4ZumxhDQCqVxVVjUwdlUNayuD/RL4wdwzJSdJdC3D6A7QP4HD5WWlMKc5OiHyAAy1ufv7iBo4ZP5yvhGFa7vLSAg60drKlrjnk5w7GltpmfvFSJSdPKeTrnykJ6lwZqck8tKiMiXa2cGV18H0eGgDUEYwxrK/ufQ0AfxTlpHPq0UU8t7oKj9cMyTxAsaqspIAVOw9EzTfX9VUN3SvAhdIdL2+gsa2TX1402+9RZYOxwF4oPpqGg7q7vHz3n6sZlprc58pmgzU8M41H7WzhKx5eHvT5NACoI1Q3tHOwtXPQHcC+vjRvHPsa2/lgq+vQTKBaAzjC/JJ8mtq7+LS2KdJFYV9DO5f9+UMu+uP7vLXpyFyOQH2w1cWTK/dw9cmTmD46sC8VAxlfMIxRuRlR1Q9w7+ufsr6qkV9eNDukTV6+2cLB0gCgjnBoDeDA/1nPmF5MTkYKz6zaQ32TMw2E1gB6chLCIj0vkDGGW55bh7vLS+mILK55bGVIgkBHl4cfP7eO8QXDuP6MKSEoae9EhPLSApZvd0XFNNsfbXPxR3uOo3Nm+Te99WAcPTKHh66YH/R5NACoI1RUNZCcJEF9W8tITeb8OWN4Zf0+du1vI0nQxbt7MS5/GCNz0yPeD7Dkk2pe31DLDWdP5Z/fOJ4pxdkhCQJ/fGsr2+pa+MWFs8M+Dcj8Umua7V39rEsxFBraOvn+4k+YUJDJrZ8P3WinnpwvD8HQAKCOUFHdyOSirEEPV+vpS/PG0tbp4ZnVeyjISg9L22+sExGrHyCCNQBXcwe3vVDJ3PHDufKkUoZnpvHE1QuCDgJbapu5b9lWvjB3DKceXRTiUh9pQWl05APc+vx69jW2c8+XjwlJols4DRgARGS8iCwTkQ0iUiEi19vbC0RkqYhstn/n+xxzs4hsEZFNIvLZcL4AFXrrqxv6XAN4MI6bmM+EgkwOtnZqElg/5k/Mp+pgG1UHI7NAzK1LKmhq7+Tui+d0B+nDgsBfV7JskEHAGMOPn11HRmpSwDN9DtZRRdnkZ6ZGNAC8ubGG59ZU8+3Tj2LehPyBD4gwf2oAXcD/M8ZMB44HrhORGcBNwBvGmCnAG/Z97McuBWYC5wD3iYhOARkjapvaqWnsCKoD2OEsFwk6Aqg/ZRFcKP61in28uHYv3z59CkePPDznwwkCR4/K5huDDAJPrtzDR9v3c/N50ynKGZrO/6QkYX5JAcsj2Jz27OpqRmSl8a3TjopYGQZjwABgjNlrjFll324CNgBjgQuAR+3dHgUutG9fAPzDGNNhjNkObAHKQ11wFR6BZgD35aJjreluNQu4b9NG5ZCdnjLkzUANrZ3c8tx6po3K4dqFk3vdZ3hmGo9f5RMENg4cBFzNHdzx8gbml+Tz5bLxoS52v8pLC9jpamVfQ/uQPi9Ywz7f2lTLGdOLSUmOjdb1QZVSREqAY4GPgJHGmL1gBQnAWYVjLLDb57A99jYVA5zkkhkhCgATRmTyvTOP7q4JqCOlJCdx7IThQ54R/IuXKtnf4ubui+eS2s8F67Ag8NjAQeAXL22gpaOLO744OyRj3wfDyQeIRC1g+fb9NLV3cdaMUUP+3IHyOwCISDbwNPBdY0x/KWi9feJHjMsSkWtEZIWIrKirq/O3GCrM1lc1UDIik9wAF+jozfVnTmFhlK/SFWnzSwrYVNNEQ1vnkDzfO5/W8eTKPVxzyiRmjxu4uW94ZhpPXHV8dxB4c2NNr/u9t7meZ1dX8c1TJzNl5OCnEQnW9NFWbWr59qFbWN2xtHIfGalJnHRU4OsbDDW/AoCIpGJd/J8wxjxjb64RkdH246MB52vBHsC33jcOOHxqSMAY84AxpswYU1ZUFP4RAso/VgZw8O3/anDKSvIxBlbtCn8zUHNHFzc/s45JRVmDGpufl5nKE1cdz9RROXzzsVVHBIH2TmvMf8mITK6LUBt4SnISx03MH/KOYGMMSytrOOmoopha9c6fUUACPARsMMb81uehJcAi+/Yi4Hmf7ZeKSLqIlAJTgOBzllXYNbR2snt/GzPHhidbU/XtmPHDSUmSIekI/tWrG6luaOPui+cMeqhvXmYqj1+1oDsIvLHhUBD4/Ztb2Olq5fYvzg56CHEwyksL+LRmaJfbrNzbSHVDO2fPCM0U10PFnxrAicDXgNNFZI39cx5wJ3CWiGwGzrLvY4ypABYDlcCrwHXGmMSZ8DyGVex1MoC1BjDUMtNSmDk2L+wZwR9tc/FXeynG4yYGlkjkGwSufdwKAp/WNHH/21u5aN5YToxwE0h5qZNdPXS1gKWVNYjA6dNjq6lzwCwFY8x79N6uD3BGH8fcDtweRLlUBFRUhXYEkBqc+RPzeezDnXR0eUhPCf036PZODzc9Y03LEOxSjE4Q+NpfPuKbj69kfEEmORkp3PK5oRnz35854/JIS0li+fb9nD1zaDpkl1bWcNyE/Jib7yo2xiqpIVFR3cDovAxdvD1CykoK6Ojysr4qPEsb3rP0U7bXt3DnRXPITAs+QzUvM5XHrlzA9NG5bKtr4cefm0FBFMz3lJ6SzLHjhw/ZSKCqg21UVDdyZow1/4AGAOVjfXWjdgBHUFmJlTkajn6AT3Yf5M/vbuOy8vEhbaLJy0zl8asX8PAV8/lSFA31XVBawPqqBpo7usL+XE4/yFkaAFSsanV3sbWumVnaARwxhdnpTCrMCnk/QEeXhx889QnFORncfN70kJ4bIDcjldOmFWONF4kO5aUj8BpYuTP8o6qWVtYwqSiLyUXZYX+uUNMAoADYsLcRY9AaQISVleSzcmdolzb8w7KtfFrTzB0XzQppfkc0mzfRGlUV7nyAxvZOPtzmislv/6ABQNmcKSC0BhBZZSXW0obb6kOztOGGvY3ct2wLFx4zhtOnxeZFKhCZaSnMGpsX9nyAtzbV0ekxnDU9Nt9bDQAKsDKAR2SlMSrEi3WrwQnlAjFdHi8/fGotwzNTwzovfbRaUFrAJ7sbQrJyVl9er6xhRFYax8bAzJ+90QCgAFhf1ciMMblR1Y6biEpGZFKYnRaSMex/fnc766oauO0Ls8iPgtE5Q628tAC3x8snuw+G5fydHi/L7MnfYnWtCw0Aio4uD5trm5gVgimgVXBEhLKJwS8Qs2FvI/e8/innzBzFebNjZ3KyUCqbWIBI+BaI+WibNfnbmTHa/AMaABSwuaaZTo/RDOAoUVaSz679rdQ0BjalcUNbJ9c+vpLhw1L5+YWzErZWl5eZytSROWHLB3Amfzt5SuzOZaYBQB1aBF47gKPC/O4FYgZfC/B6DTc8+Ql7DrRx31fnDdliLNFqQWkBK3ceoNPjDel5jTG8vqE25iZ/60kDgGJ9dQM56SmMz8+MdFEU1loMw1KTA+oH+NM721haWcOPzpvevdJYIisvHUGr29M9yi1UKvc2UnWwLeYmf+tJA4CiotrqAB7qxTtU71IDXCDm/S313P3aRs6fM5qvn1gSnsLFmPml1uicUOcDOJO/nTYttiZ/60kDQILr8njZsLdRO4CjTFlJARv2NtLU7t8CMXsb2vj231czqSibu740J2Hb/XsqzslgUmFWyDuCl1bWMG9Cfsw3sWkASHDb6lto7/TqDKBRZn5JPl4Dq3cNPITR3eXlv59YRXunh/svP46s9OAneosn5aUFfLzjQMiyq6vtyd9iNfvXlwaABFdR7XQAaw0gmhw7IZ8k8W9iuDte3sDqXQe5+5K5HFUce/PRhFt5aQENbZ1s2BeafoDXY3jyt540ACS49VWNZKQmMakwK9JFUT6y01OYMSZ3wIzg59dU8cj7O7j6pFLOmz16iEoXW06aUsiw1GTufm0TxgRfC1haWcOkwtic/K0nDQAJbn1VA9NG5ZKSrH8K0aZsYgGrd/c9hHHTviZuenod80vyufHcaUNcuthRnJPBD8+Zylub6nh6VVVQ54r1yd960v/6BOb1GiqrG3X8f5SaX1JAe6e31yGMTe1Wsld2Rgp/+Mo8UjWA92vRCSXML8nnZy9UBJxgB/C2M/mbBgAV63YfaKWpo0szgKNUXwvEGGP4wZNr2bm/lT98ZR7FOoHfgJKShF9dPJeOLi8/fnZdwE1BS2N88reeNAAkMGfpQe0Ajk4jczOYUJB5RD7An9/dxqsV+7j53GndC6CrgZUWZnHD2VN5fUMtz6+pHvTxzuRvp0+L3cnfetIAkMDWVzeQkiRMGRn7nVnxqqwknxU7DnR/Y/1wm4u7Xt3EebNHcdVJpREuXey58qRSjp0wnJ++UEFt0+CagpZvtyZ/i5fmH9AAkNAqqhs5emQO6SmxO5dJvJtfUoCrxc32+hZqGtv51t9WM3FEJr+6eK4mewUgOUm4++K5tLo9/OS5ikE1BS2trIn5yd960gCQoIwxVFQ1aAdwlJtv9wN8sM3FdU+sotXdxZ8uP45sTfYK2FHF2XzvzKN5tWIfL63b69cxxhiWVtbE/ORvPWkASFD7Gttxtbh1DeAoN7kom/zMVO54aQMrdh7gri/NYcrInEgXK+b918mlzB2Xx0+er8DV3DHg/s7kb2fNiO25f3rSAJCgDnUAaw0gmokIZSUFtLg9fP3EEj4/d0ykixQXUpKT+NXFc2lu7+InSyoG3P/1ylpEiLt1lTUAJKiK6gZEYPpoDQDR7qsLJvDlsvHcfO70SBclrkwdlcN3zjiKl9bu5dX1/TcFLd2wLy4mf+tJA0CCae/0cP/bW3nove1MG5VLZpq2JUe7hVOLueviOaSl6L9rqH3j1MnMHJPLLc+t50CLu9d9qg+2sb4qPiZ/60n/ohKEx2t4euUeTv/1W9z5ykbmlxTwh68cG+liKRVRqclJ3H3xXA62dnLbC703BTmTv8Xy2r990a9/CeCdT+v45Ssb2bC3kTnj8vj1f8zlM5MLI10spaLCjDG5fOv0o7j39c18bs6YI77pO5O/xeNMqxoA4lhFdQN3vrKRdzfXM75gGP972bGcP3u0rvylVA//vfAoXl2/jx8/u47ykgLyMlOBQ5O/XXlifCbdaRNQHKo62Mb3F6/h/P97j3VVDfzP+TN4/fun8oW5Y/Tir1Qv0lKS+PUlc3G1uPnZi5Xd2+Nt8reetAYQRxraOrnvrS08/O8dAHzjlMlcu3AyecNSI1swpWLArLF5XHvqZH6/bAvnzxnNadOK427yt540AMSBji4Pj32wk98v20JDWycXHTuO7599NGOHD4t00ZSKKd8+4yj+VbmPm59Zx8vXn8yyTbWcM3NU3Ez+1pMGgBjT5fGyw9XCxn1NbNzbxMZ9TXyy5yB1TR2ccnQRN50zjRm6vq9SAUlPSebui+fyxfv+zdce+ijuJn/rKWwBQETOAX4HJAMPGmPuDNdzxSNjDHVNHWzc18SmfU1s2NfIpn1NbK5txt1lrRCVnCSUFmaxoLSAL88fH1eTVCkVKXPHD+eaUyZz/9tbSU+Jr8nfegpLABCRZOAPwFnAHuBjEVlijKns/8jgebwGV0sHtY0d1DVZP7VN7dQ2WdvaOj2kpySRnpps/U5JIqP7djLpqUduS00WvMY6d5fXa/82h357vHT5bvMYPF4vgSw50dTexaZ9TWyqaWK/T2JKcU46U0flcMVnSpg6Moepo3I4qjibjNT4mZhKqWjx3TOnsGxjLVNGZsfV5G89hasGUA5sMcZsAxCRfwAXAL0GgE37mjj7nretC7DvxTfVugBn2L+dC3Z6ajJpyUk0tHVS29RuX+StH1dzB95errx5w1IpzkknMy2Zji4v7i4vHV1e2js9dHR56ejy0OkJfsFoX4E0G2akJjNlZA5nzxjJ1FHWhX7aqFwKstJCWjalVN8yUpN5/lsnxm3bvyNcAWAssNvn/h5gQV87Z6YlM7kou/uC3Nbp4WCbm45OL+1dHjo6vd0X6fbOQwtkJycJhdlpFOWkMzI3g9lj8yjKSac4J52inAyKc9Mpyk6nKCfdr2/KHq+ho8fzdXR56ej0kpwkpCSL9TvJ+Z106H6ykOpzX4dbKhXbEqF2Ha4A0NvV77Cv1yJyDXANwIQJE/jj5cf5dWJjDG6PdYHOSksJaYROThIy01LI1C/bSqkEEK5EsD3AeJ/744DDFuE0xjxgjCkzxpQVFfnfySIipKckk5uRGvfVM6WUCqdwBYCPgSkiUioiacClwJIwPZdSSqkAhKUJyBjTJSLfAl7DGgb6F2PMwKsuKKWUGjJhywMwxrwMvByu8yullAqOTganlFIJSgOAUkolKA0ASimVoDQAKKVUghJjQjv9QUCFEGkCNgVwaB7QMATHABQC9UPwXIGWb6ieK5D3IdDnGsr3Yqg+30CP0/ci+OeKx//hqcaYnACey2KMifgPsCLA4x4YimMCLeMQl29InitGPqsheV3R/lnpexGZ1xXtn5XvT6w3Ab0wRMcEaijLp+9F8McN1fPoexHccfp3GyLR0gS0whhTFuly9CcWyjgU4vV9iNfXFYh4fS/i8XUF+5qipQbwQKQL4IdYKONQiNf3IV5fVyDi9b2Ix9cV1GuKihqAUkqpoRctNQCllFJDTAOAUkolKA0ANhEZJyLPi8hmEdkqIr+zp7Lua//vikjmUJZxKIiIEZHf+Ny/QUR+GsEiBU1EPCKyRkQqROQTEfm+iCT8376INEe6DKHi8xk7PyX97LtQRF4cutIFzv5/fMznfoqI1IWq/An/TwAgIgI8AzxnjJkCHA1kA7f3c9h3gbgLAEAHcJGIFEa6ICHUZow5xhgzEzgLOA+4NcJlUqHlfMbOz45IFyhEWoBZIjLMvn8WUDWYE4hIn7M+awCwnA60G2MeBjDGeIDvAVeKSJaI/FpE1onIWhH5toh8BxgDLBORZREsdzh0YY0s+F7PB0Rkooi8Yb8Pb4jIBBHJE5EdzjdqEckUkd0ikjrUBfeHMaYWaynSb4klWUTuFpGP7df1DWdfEfmh/bl/IiJ3Rq7U4SMi2fZnucp+rRfY20tEZIOI/NmuOf3L5yIUE/r7bIFcEXlWRCpF5P4orxG+AnzOvn0Z8HfnAREpF5H3RWS1/Xuqvf0KEXlSRF4A/tXnmYPJIouXH+A7wD29bF8NXA88DaTY2wrs3zuAwkiXPQzvRTOQa7++POAG4Kf2Yy8Ai+zbV2LVmACeB06zb38ZeDDSr6Pna+pl2wFgJFYwuMXelg6sAEqBc4H3gUzfzz2efuzPOgXIte8XAluw1vQuwfoycIz92GLg8kiXuZ/X4gHW2D/P2tv6+mwXAu3AJKwFq5YCF0f6NfTzGc0BngIy7Ne3EHjRfjzX59p0JvC0ffsKrKV5+/27DduCMDFG6LFovc/2U4D7jTFdAMaY/UNZsEgwxjSKyF+xAmObz0MnABfZtx8DfmXf/ifWhX8Z1vKf9w1RUYPhLCh9NjBHRC627+cBU7D+mR42xrRCXH/uAtwhIqcAXmAsVmAE2G6MWWPfXokVFKJVmzHmmB7b+vps3cByY8w2ABH5O3AS1kU26hhj1tp9Gpdx5CJbecCjIjIF6xrmW/NeOtDfbTRXe4ZSBXBYNp2I5GItbN9XcIh39wJXAVn97OO8L0uAc0WkADgOeDPMZQuKiEzC+sZYi/X5ftscajsuNcb8i8T53L8KFAHH2RfQGqxvmmD1Bzk8hHEFwTDp67OFIz/baP+slwC/xqf5x/ZzYJkxZhbweQ59dmD1H/RLA4DlDSBTRP4TrLZD4DfAI1jtZ990OlLsixxAExD4LHxRzv7msBgrCDjex/qGD9aF4z1732ZgOfA7rKqpZwiLOigiUgTcD/zeWHXl14BrnT4LETlaRLKwPvcrnZFePp97vMkDao0xnSJyGjAx0gUKob4+W4ByESm12/6/jP23HMX+AvzMGLOux/Y8DnUKXzHYk2oAAOwLwReBS0RkM/ApVhvhj4AHgV3AWhH5BPiKfdgDwCtx2Ans6zdY7cKO7wBfF5G1wNew+kcc/wQut39Hm2HOMFDgdayL+232Yw8ClcAqEVkP/AmrTfVVrG9dK0RkDVZfSNywv9B0AE8AZSKyAiuob4xowUKr18/WfuwD4E5gPbAdeDYiJfSTMWaPMeZ3vTz0K+CXIvJvrP6MQdGpIJRKQCIyF/izMaY80mVRkaM1AKUSjIh8E6st+ZZIl0VFltYAlFIqQWkNQKk4JyLjRWSZndhVISLX29sLRGSpWNOfLBWRfHv7CHv/ZhH5fY9zXeaTFPlqnGWMJxytASgV50RkNDDaGLNKRHKwxvRfiDVqZL8x5k4RuQnIN8bcaI+UORaYBcwyxnzLPk8KUA3MMMbUi8ivgFZjzE+H/lWpUNAagFJxzhiz1xizyr7dBGzASvi6AHjU3u1RrKCAMabFGPMe1kg4X2L/ZImIYGWhVof/FahwibXEDqVUEOyM0mOBj4CRxpi9YAUJESnu71g7V+BaYB1WktFm4LqwFliFldYAlEoQIpKNNa/Vd40xjQEcnwpcixVAxgBrgZtDWkg1pDQAKJUA7Iv308ATxphn7M01dv+A009QO8BpjgEwxmy1kycXA58JU5HVENAAoFScs9vrHwI2GGN+6/PQEmCRfXsR1qyu/akCZtjTaYA1N/2GUJZVDS0dBaRUnBORk4B3sdruvfbmH2H1AywGJmBNd3KJM3ukiOzA6uRNAw4CZxtjKu0ksuuBTmAncIUxxjV0r0aFkgYApZRKUNoEpJRSCUoDgFJKJSgNAEoplaA0ACilVILSAKCUUglKA4BSQRCRt0SkbOA9lYo+GgCUUipBaQBQCUVEfigi37Fv3yMib9q3zxCRx0XkbBH5QERWiciT9vw5iMhxIvK2iKwUkdecKRR8zpskIo+KyC+G/lUpFRgNACrRvAOcbN8uA7LteXJOwsqUvQU40xgzD1gBfN9+/P+Ai40xxwF/AW73OWcK1uLqnxpjdJlFFTN0OmiVaFYCx9kLo3QAq7ACwclYc+PMAP5tTZ9DGvABMBVrcZSl9vZkYK/POf8ELDbG+AYFpaKeBgCVUOw57XcAXwfex5rS+DRgMrAdWGqMucz3GBGZDVQYY07o47TvA6eJyG+MMT0XUVEqamkTkEpE7wA32L/fBb4JrAE+BE4UkaMARCRTRI4GNgFFInKCvT1VRGb6nO8h4GXgSXvZRKViggYAlYjeBUYDHxhjarCWPnzXGFOHtU7u30VkLVZAmGaMcQMXA3eJyCdYweKwefDtaZZXAY+JiP5fqZigs4EqpVSC0m8qSimVoDQAKKVUgtIAoJRSCUoDgFJKJSgNAEoplaA0ACilVILSAKCUUglKA4BSSiWo/w+IKgH2X2jYNgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.groupby(\"week\").title.count().plot(title=\"Number of articles per Week\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Number of words per article"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     5000.000000\n",
       "mean       908.503800\n",
       "std       1744.611774\n",
       "min         29.000000\n",
       "25%        298.000000\n",
       "50%        568.000000\n",
       "75%       1086.000000\n",
       "max      94980.000000\n",
       "Name: text, dtype: float64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.text.str.split().apply(len).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1c2614759c8>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAD4CAYAAADGmmByAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAATL0lEQVR4nO3df7AlZ13n8feHEMOvKGQzicMkszdQs7iBwhCvWay4u1HExCCMbIkOtYspCx2rDCWUVMkELcE/pipuSfxRLMoAKQMCMQhIlsHVIYIUVcgwCcHMZBIzmBGGmcrEX5WobGLC1z9OX+ZkuHPvudOnzzl9z/tVdep2P6e77/c+SZ3PPN1P90lVIUnS6XrStAuQJPWbQSJJasUgkSS1YpBIkloxSCRJrTx52gW0ce6559bCwsK0y5CkXrn99tv/rqo2jOt4vQ6ShYUF9u3bN+0yJKlXkvztOI/nqS1JUisGiSSpFYNEktSKQSJJasUgkSS1YpBIkloxSCRJrRgkkqRWDBJJUitzHSQLO3azsGP3tMuQpF7rLEiSXJjkU0kOJjmQ5PVN+1uTfC3Jnc3r6qF9rktyKMm9Sa7sqjZJ0vh0+aytx4A3VtUdSc4Gbk+yp3nvN6vqN4Y3TnIxsA14PvBs4JNJ/lNVPd5hjZKkljobkVTVsaq6o1l+GDgIbFphl63AzVX1SFXdDxwCLuuqPknSeEzkGkmSBeBFwOebptcl+askNyZ5VtO2Cfjq0G5HWCZ4kmxPsi/JvgcffLDDqiVJo+g8SJI8A/gw8Iaqegj4XeC5wCXAMeBtS5sus3t9S0PVrqparKrFDRvG9jh9SdJp6jRIkpzJIETeX1UfAaiqB6rq8ar6BvAuTpy+OgJcOLT7BcDRLuuTJLXX5aytAO8BDlbVDUPtG4c2eyWwv1m+FdiW5KwkFwFbgL1d1SdJGo8uZ21dDrwGuCvJnU3bm4FXJ7mEwWmrw8DPAVTVgSS3AHczmPF1rTO2JGn2dRYkVfVZlr/u8YkV9tkJ7OyqJknS+M31ne2SpPYMEklSKwaJJKkVgwR8cKMktWCQSJJaMUgkSa0YJJKkVgwSSVIrBokkqRWDRJLUikEiSWrFIJEktWKQSJJaMUgkSa0YJJKkVgwSSVIrBokkqRWDRJLUikEiSWrFIJEktWKQSJJaMUgkSa0YJJKkVgwSSVIrBokkqRWDRJLUikHSWNixm4Udu6ddhiT1jkEiSWrFIJEktWKQSJJaMUgkSa10FiRJLkzyqSQHkxxI8vqm/Zwke5Lc1/x81tA+1yU5lOTeJFd2VZskaXy6HJE8Bryxqv4z8GLg2iQXAzuA26pqC3Bbs07z3jbg+cBVwDuSnNFhfZKkMegsSKrqWFXd0Sw/DBwENgFbgZuazW4CfqxZ3grcXFWPVNX9wCHgsq7qkySNx0SukSRZAF4EfB44v6qOwSBsgPOazTYBXx3a7UjTdvKxtifZl2Tfgw8+2GXZkqQRdB4kSZ4BfBh4Q1U9tNKmy7TVtzRU7aqqxapa3LBhw7jK/CZvTJSktek0SJKcySBE3l9VH2maH0iysXl/I3C8aT8CXDi0+wXA0S7rkyS11+WsrQDvAQ5W1Q1Db90KXNMsXwN8bKh9W5KzklwEbAH2dlWfJGk8ntzhsS8HXgPcleTOpu3NwPXALUleC3wFeBVAVR1IcgtwN4MZX9dW1eMd1idJGoPOgqSqPsvy1z0AXnKKfXYCO7uqSZI0ft7ZLklqxSCRJLVikEiSWunyYvvM8j4RSRofRySSpFYMEklSKwaJJKkVg0SS1IpBIklqxSCRJLVikEiSWjFIJEmtGCSSpFYMEklSKwaJJKkVg0SS1IpBIklqxSCRJLVikEiSWjFIJEmtGCSSpFYMEklSKwaJJKkVg0SS1MpIQZLkBV0XIknqp1FHJL+XZG+Sn0/yzE4rkiT1ykhBUlXfD/xP4EJgX5IPJHlpp5VJknph5GskVXUf8CvAm4D/DvxOknuS/I+uipMkzb4nj7JRkhcCPw28DNgDvLyq7kjybOBzwEe6K3F8FnbsnnYJkrTujBQkwNuBdwFvrqqvLzVW1dEkv9JJZZKkXhg1SK4Gvl5VjwMkeRLwlKr616p6X2fVSZJm3qjXSD4JPHVo/WlNmyRpzo0aJE+pqn9eWmmWn7bSDkluTHI8yf6htrcm+VqSO5vX1UPvXZfkUJJ7k1y51j9EkjQdowbJvyS5dGklyfcAX19he4DfB65apv03q+qS5vWJ5ngXA9uA5zf7vCPJGSPWJkmaolGvkbwB+FCSo836RuAnV9qhqj6TZGHE428Fbq6qR4D7kxwCLmMwI0ySNMNGCpKq+kKS7wKeBwS4p6r+7TR/5+uS/BSwD3hjVf0jsAn4y6FtjjRt3yLJdmA7wObNm0+zBEnSuKzloY3fC7wQeBHw6iYM1up3gecClwDHgLc17Vlm21ruAFW1q6oWq2pxw4YNp1GCJGmcRr0h8X0MAuBO4PGmuYD3ruWXVdUDQ8d8F/DxZvUIg8evLLkAOMoULd28ePj6l02zDEmaeaNeI1kELq6qZUcJo0qysaqONauvBJZmdN0KfCDJDcCzgS3A3ja/S5I0GaMGyX7gOxmcjhpJkg8CVwDnJjkCvAW4IsklDEYzh4GfA6iqA0luAe4GHgOuXbr5UZI020YNknOBu5PsBR5ZaqyqV5xqh6p69TLN71lh+53AzhHrWROfsSVJ3Rk1SN7aZRGSpP4adfrvXyT5j8CWqvpkkqcB3jAoSRr5q3Z/Fvgj4J1N0ybgj7sqSpLUH6PeR3ItcDnwEHzzS67O66ooSVJ/jBokj1TVo0srSZ7MKW4YlCTNl1GD5C+SvBl4avNd7R8C/m93ZUmS+mLUWVs7gNcCdzG49+MTwLu7KmqWnDx12DvdJemJRp219Q0GX7X7rm7LkST1zajP2rqfZa6JVNVzxl6RJKlX1vKsrSVPAV4FnDP+ciRJfTPSxfaq+vuh19eq6reAH+y4NklSD4x6auvSodUnMRihnN1JRZKkXhn11NbbhpYfY/Dk3p8YezWSpN4ZddbWD3RdSN/4xVeSNDDqqa1fXOn9qrphPOVIkvpmLbO2vpfBNxkCvBz4DPDVLoqSJPXHWr7Y6tKqehggyVuBD1XVz3RVmCSpH0Z91tZm4NGh9UeBhbFXI0nqnVFHJO8D9ib5KIM73F8JvLezqiRJvTHqrK2dSf4E+K9N009X1Re7K0uS1BejntoCeBrwUFX9NnAkyUUd1SRJ6pFRv2r3LcCbgOuapjOBP+iqKElSf4w6Inkl8ArgXwCq6ig+IkWSxOhB8mhVFc2j5JM8vbuSJEl9MmqQ3JLkncAzk/ws8En8kitJEqPP2vqN5rvaHwKeB/xqVe3ptDJJUi+sGiRJzgD+tKp+CDA8JElPsOqprap6HPjXJN8xgXp6b2HH7m8+GViS5sGod7b/f+CuJHtoZm4BVNUvdFKVJKk3Rg2S3c1LkqQnWDFIkmyuqq9U1U2TKkiS1C+rXSP546WFJB9ey4GT3JjkeJL9Q23nJNmT5L7m57OG3rsuyaEk9ya5ci2/S5I0PasFSYaWn7PGY/8+cNVJbTuA26pqC3Bbs06Si4FtwPObfd7RzBaTJM241a6R1CmWV1VVn0mycFLzVuCKZvkm4NMMnuG1Fbi5qh4B7k9yCLgM+NxafuckOCNLkp5otSD57iQPMRiZPLVZplmvqvr2Nf6+86vqGIOdjyU5r2nfBPzl0HZHmjZJ0oxbMUiqalKnl7JM27IjoCTbge0Amzdv7rImSdIIRp3+Oy4PJNnYjEY2Aseb9iPAhUPbXQAcXe4AVbUL2AWwuLi4ptNtXfKUl6R5tZYvthqHW4FrmuVrgI8NtW9LclbzhVlbgL0Trk2SdBo6G5Ek+SCDC+vnJjkCvAW4nsGThF8LfAV4FUBVHUhyC3A38BhwbfNoFknSjOssSKrq1ad46yWn2H4nsLOreiZtYcduDl//smmXIUmdm/SpLUnSOmOQSJJaMUgkSa0YJJKkViZ9H8m64/0jkuadI5IZ5rctSuoDg0SS1IpBIklqZV1fI/G0kCR1zxFJz3jdRNKsMUgkSa0YJJKkVgySDnkaStI8MEgkSa0YJJKkVgySCRj1FJenwiT1kUEiSWplXd+Q2BfjGIUsHcNvZZQ0aY5IJEmtGCSSpFYMEklSK14jmQJnZklaTwySCRo1QAwaSX3iqS1JUisGSY84UpE0izy11QPeZyJpljkikSS1YpBIkloxSCRJrRgkkqRWDJJ1xkfRS5o0g0SS1MpUpv8mOQw8DDwOPFZVi0nOAf4QWAAOAz9RVf84jfr6ZLXRh6MTSV2b5ojkB6rqkqpabNZ3ALdV1RbgtmZdkjTjZumGxK3AFc3yTcCngTdNq5hZ50hD0qyYVpAU8GdJCnhnVe0Czq+qYwBVdSzJecvtmGQ7sB1g8+bNk6q3dwwaSZMyrSC5vKqONmGxJ8k9o+7YhM4ugMXFxeqqQEnSaKZyjaSqjjY/jwMfBS4DHkiyEaD5eXwatUmS1mbiQZLk6UnOXloGfhjYD9wKXNNsdg3wsUnXJklau2mMSM4HPpvkS8BeYHdV/T/geuClSe4DXtqsq2NeS5HU1sSvkVTV3wDfvUz73wMvmXQ98hHzktrxznZJUisGiSSpFYNkzvhQR0njNkt3tmtGnBw0XjuRtBJHJJKkVtbliMRTN6uzjySNiyMSSVIrBokkqRWDRGt28swvZ4JJ821dXiPReBkSklayroLED7x27D9Jp8NTWxobT3FJ88kg0dgZJtJ8MUg0UxzVSP2zrq6RaLJW+sD30fTS/HBEok45wpDWP4NEktSKQSJJasUgkSS1si4utnsOvn9Ovhi/2n/DhR27vXAvzah1ESTqL/8RIPWfp7Y0k5ztJfWHIxJNlOEgrT+OSDQ3HOVI3TBINHcMFGm8PLWliejyg9vHsUjTZZBopi0XQKsFh8EiTZZBot46OWROZ9QzS6FkAKqvDBL1xumeHjvVfqM8vXi190/+0B++cdJg0LwwSLRutRmhrPVYXrzXPDNIJCYTBKOOUFZ6fMwsnILrQt/rX4v1+LcaJFILowTQOENqPX4Iqf9mLkiSXAX8NnAG8O6qun7KJUlrcrrXV9Z6nLa/61Qjn9XWTzZ87FG2Wasuw/Pker2+dXpmKkiSnAH8H+ClwBHgC0luraq7p1uZNH6TvK6yUiisNvttHMG41qc9j8NKEyJW22eSTie0Zi3oZipIgMuAQ1X1NwBJbga2AgaJNOR0JwV0aRy/+3Q+5Ef9MB3H5ItTBeFyo7LVQnMtEzjWem1slFmF45SqGvtBT1eSHweuqqqfadZfA/yXqnrd0Dbbge3N6guA/RMvdDadC/zdtIuYEfbFCfbFCfbFCc+rqrPHdbBZG5FkmbYnJF1V7QJ2ASTZV1WLkyhs1tkXJ9gXJ9gXJ9gXJyTZN87jzdpDG48AFw6tXwAcnVItkqQRzFqQfAHYkuSiJN8GbANunXJNkqQVzNSprap6LMnrgD9lMP33xqo6sMIuuyZTWS/YFyfYFyfYFyfYFyeMtS9m6mK7JKl/Zu3UliSpZwwSSVIrvQ2SJFcluTfJoSQ7pl1PF5LcmOR4kv1Dbeck2ZPkvubns4beu67pj3uTXDnU/j1J7mre+50ky02znllJLkzyqSQHkxxI8vqmfR774ilJ9ib5UtMXv9a0z11fLElyRpIvJvl4sz6XfZHkcPM33Lk0vXdifVFVvXsxuBD/ZeA5wLcBXwIunnZdHfyd/w24FNg/1Pa/gR3N8g7g15vli5t+OAu4qOmfM5r39gLfx+A+nT8BfmTaf9sa+2EjcGmzfDbw183fO499EeAZzfKZwOeBF89jXwz1yS8CHwA+3qzPZV8Ah4FzT2qbSF/0dUTyzUepVNWjwNKjVNaVqvoM8A8nNW8FbmqWbwJ+bKj95qp6pKruBw4BlyXZCHx7VX2uBv+XvHdon16oqmNVdUez/DBwENjEfPZFVdU/N6tnNq9iDvsCIMkFwMuAdw81z2VfnMJE+qKvQbIJ+OrQ+pGmbR6cX1XHYPABC5zXtJ+qTzY1yye391KSBeBFDP4lPpd90ZzKuRM4DuypqrntC+C3gF8CvjHUNq99UcCfJbm9eZQUTKgvZuo+kjVY9VEqc+hUfbJu+irJM4APA2+oqodWOHW7rvuiqh4HLknyTOCjSV6wwubrti+S/ChwvKpuT3LFKLss07Yu+qJxeVUdTXIesCfJPStsO9a+6OuIZJ4fpfJAM/yk+Xm8aT9Vnxxplk9u75UkZzIIkfdX1Uea5rnsiyVV9U/Ap4GrmM++uBx4RZLDDE5v/2CSP2A++4KqOtr8PA58lMElgIn0RV+DZJ4fpXIrcE2zfA3wsaH2bUnOSnIRsAXY2wxnH07y4mb2xU8N7dMLTd3vAQ5W1Q1Db81jX2xoRiIkeSrwQ8A9zGFfVNV1VXVBVS0w+Az486r6X8xhXyR5epKzl5aBH2bwZPTJ9MW0Zxq0mKFwNYPZO18Gfnna9XT0N34QOAb8G4N/KbwW+A/AbcB9zc9zhrb/5aY/7mVopgWw2PxP9WXg7TRPNOjLC/h+BsPrvwLubF5Xz2lfvBD4YtMX+4Ffbdrnri9O6pcrODFra+76gsEM1i81rwNLn4mT6gsfkSJJaqWvp7YkSTPCIJEktWKQSJJaMUgkSa0YJJKkVgwSSVIrBokkqZV/Bzi0pvKyCq4WAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.text.str.split().apply(len).plot(kind=\"hist\", bins=3330, xlim=(0,5000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Clean Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 443 ms\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Index</th>\n",
       "      <th>date</th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "      <th>week</th>\n",
       "      <th>clean_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>621031</td>\n",
       "      <td>2017-10-26 15:00:54</td>\n",
       "      <td>eBay launches visual search tools that let you shop using photos from your phone or web0eBay today is launching two new visual search tools that will allow online shoppers to use photos they snap,...</td>\n",
       "      <td>eBay launches visual search tools that let you shop using photos from your phone or web</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>ebay launches visual search tools that let you shop using photos from your phone or webebay today is launching two new visual search tools that will allow online shoppers to use photos they snap h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>621032</td>\n",
       "      <td>2017-10-26 00:00:00</td>\n",
       "      <td>Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller : The SaltScientists are accusing the seed-and-pesticide giant Monsanto of denying the risks of its latest weedkilling...</td>\n",
       "      <td>Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>monsanto attacks scientists after studies show trouble for its new weedkiller  the saltscientists are accusing the seedandpesticide giant monsanto of denying the risks of its latest weedkilling te...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>621035</td>\n",
       "      <td>2017-10-26 14:41:12</td>\n",
       "      <td>Hard Drive Stats for Q3 2017In Q3 2017, Backblaze introduced both 10 TB and 12 TB hard drives into our data centers, we continued to retire 3 TB and 4 TB hard drives to increase storage density, a...</td>\n",
       "      <td>2017 Hard Drive Reliability by Drive Size and Brand</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>hard drive stats for q in q  backblaze introduced both  tb and  tb hard drives into our data centers we continued to retire  tb and  tb hard drives to increase storage density and we added over  p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>621036</td>\n",
       "      <td>2017-10-26 15:27:11</td>\n",
       "      <td>This is part 1 of the “How Unreal Renders a Frame” series, you can access part 2 and part 3 as well.I was looking around the Unreal source the other day and inspired by some excellent breakdowns o...</td>\n",
       "      <td>How Unreal Renders a Frame</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>this is part  of the how unreal renders a frame series you can access part  and part  as well.i was looking around the unreal source the other day and inspired by some excellent breakdowns of how ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>621423</td>\n",
       "      <td>2017-10-26 16:02:00</td>\n",
       "      <td>Boeing builds towards its first space taxi testWhen NASA first named Boeing and SpaceX as recipients of the Commercial Crew Program, they originally aimed to end the agency's dependence on Russia ...</td>\n",
       "      <td>Boeing builds towards its first space taxi test</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>boeing builds towards its first space taxi testwhen nasa first named boeing and spacex as recipients of the commercial crew program they originally aimed to end the agencys dependence on russia by...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Index                date  \\\n",
       "0  621031 2017-10-26 15:00:54   \n",
       "1  621032 2017-10-26 00:00:00   \n",
       "2  621035 2017-10-26 14:41:12   \n",
       "3  621036 2017-10-26 15:27:11   \n",
       "4  621423 2017-10-26 16:02:00   \n",
       "\n",
       "                                                                                                                                                                                                      text  \\\n",
       "0  eBay launches visual search tools that let you shop using photos from your phone or web0eBay today is launching two new visual search tools that will allow online shoppers to use photos they snap,...   \n",
       "1  Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller : The SaltScientists are accusing the seed-and-pesticide giant Monsanto of denying the risks of its latest weedkilling...   \n",
       "2  Hard Drive Stats for Q3 2017In Q3 2017, Backblaze introduced both 10 TB and 12 TB hard drives into our data centers, we continued to retire 3 TB and 4 TB hard drives to increase storage density, a...   \n",
       "3  This is part 1 of the “How Unreal Renders a Frame” series, you can access part 2 and part 3 as well.I was looking around the Unreal source the other day and inspired by some excellent breakdowns o...   \n",
       "4  Boeing builds towards its first space taxi testWhen NASA first named Boeing and SpaceX as recipients of the Commercial Crew Program, they originally aimed to end the agency's dependence on Russia ...   \n",
       "\n",
       "                                                                                     title  \\\n",
       "0  eBay launches visual search tools that let you shop using photos from your phone or web   \n",
       "1            Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller   \n",
       "2                                      2017 Hard Drive Reliability by Drive Size and Brand   \n",
       "3                                                               How Unreal Renders a Frame   \n",
       "4                                          Boeing builds towards its first space taxi test   \n",
       "\n",
       "        week  \\\n",
       "0 2017-10-29   \n",
       "1 2017-10-29   \n",
       "2 2017-10-29   \n",
       "3 2017-10-29   \n",
       "4 2017-10-29   \n",
       "\n",
       "                                                                                                                                                                                                clean_text  \n",
       "0  ebay launches visual search tools that let you shop using photos from your phone or webebay today is launching two new visual search tools that will allow online shoppers to use photos they snap h...  \n",
       "1  monsanto attacks scientists after studies show trouble for its new weedkiller  the saltscientists are accusing the seedandpesticide giant monsanto of denying the risks of its latest weedkilling te...  \n",
       "2  hard drive stats for q in q  backblaze introduced both  tb and  tb hard drives into our data centers we continued to retire  tb and  tb hard drives to increase storage density and we added over  p...  \n",
       "3  this is part  of the how unreal renders a frame series you can access part  and part  as well.i was looking around the unreal source the other day and inspired by some excellent breakdowns of how ...  \n",
       "4  boeing builds towards its first space taxi testwhen nasa first named boeing and spacex as recipients of the commercial crew program they originally aimed to end the agencys dependence on russia by...  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re # regular expression library\n",
    "\n",
    "# get rid of everything except letters, space and .!?\n",
    "# regex explanation: https://regex101.com/r/xrfp0C/1\n",
    "# remove everything except lowercase a-z, spaces and .!?\n",
    "clean = lambda x : re.sub('[^a-z .!?]', \"\", str(x).lower())\n",
    "\n",
    "# clean text\n",
    "%time df[\"clean_text\"] = df.text.apply(clean) \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lemmatization / Stemming\n",
    "\n",
    "- Stemming -> Heuristic (fast)\n",
    "- Lemmatization -> Based on dictionary (not so fast)\n",
    "\n",
    "<img src=\"https://timkuhn.github.io/TextMining/images/lemma.png\" width=\"300\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "pip3 install -U spacy\n",
    "python3 -m spacy download en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 972 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    ebay launch visual search tool that let -PRON- shop use photo from -PRON- phone or webebay today be launch two new visual search tool that will allow online shopper to use photo -PRON- snap have s...\n",
       "1    monsanto attack scientist after study show trouble for -PRON- new weedkiller   the saltscientist be accuse the seedandpesticide giant monsanto of deny the risk of -PRON- late weedkilling technolog...\n",
       "2    hard drive stat for q in q   backblaze introduce both   tb and   tb hard drive into -PRON- datum center -PRON- continue to retire   tb and   tb hard drive to increase storage density and -PRON- ad...\n",
       "3    this be part   of the how unreal render a frame series -PRON- can access part   and part   as well.i be look around the unreal source the other day and inspire by some excellent breakdown of how p...\n",
       "4    boeing build towards -PRON- first space taxi testwhen nasa first name boeing and spacex as recipient of the commercial crew program -PRON- originally aim to end the agencys dependence on russia by...\n",
       "5    netflix will stream a strange thing aftershow for diehard fansit safe to say that the second season of netflixs strange thing be pretty eagerly anticipate and now the stream company be capitalize ...\n",
       "6    twitter block ad from two russiabacke medium agenciestwitter have announce that -PRON- will no longer allow two prominent stateowne russian medium organization from advertise on -PRON- platform . ...\n",
       "7    apnewsbreak georgia election server wipe after suit filedfile   this sept .    file photo show the screen of an electronic voting machine during testing at the kennesaw state university center for...\n",
       "8    image caption brand late book deal with -PRON- own addiction to drug alcohol sex and fameaddiction and mental health may not be the kind of issue -PRON- would normally expect to be address at a st...\n",
       "9    in this oct .    photo baby turkey stand in a poult barn at smotherman farm near waco texas . the farm be involve in a pilot project by cargills honeysuckle white brand that allow consumer to be a...\n",
       "Name: clean_text, dtype: object"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "# make sure your downloaded the english model with \"python -m spacy download en\"\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "def lemmatize(text, nlp):\n",
    "    \"\"\"Takes a text and a spacy nlp object as input and returns lemmatized tokens for the text using the spacy model\"\"\"\n",
    "    # pass to spacy \n",
    "    doc = nlp(text, disable=[\"ner\", \"pos\"])\n",
    "    \n",
    "    # get lemma for each token (i.e. word)\n",
    "    list_of_lemmatized_words = [token.lemma_ for token in doc]\n",
    "    \n",
    "    # join words to string\n",
    "    lemmatized_text = \" \".join(list_of_lemmatized_words)\n",
    "    \n",
    "    # return string\n",
    "    return lemmatized_text\n",
    "    \n",
    "\n",
    "# apply the lemmatize function to the first 50 texts\n",
    "%time df[\"clean_text\"].iloc[:10].apply(lambda text : lemmatize(text, nlp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Popularity based prefiltering\n",
    "\n",
    "Idea: Very frequent and very infrequent words are either too common or too rare to provide insights.\n",
    "\n",
    "-> removing corpus-specific stop words\n",
    "\n",
    "<img src=\"https://finnaarupnielsen.files.wordpress.com/2013/10/brownzipf.png\" width=350>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Word frequency for our corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1c21516a248>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAD4CAYAAAD2FnFTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAV+UlEQVR4nO3deZRcZ32n8edX1YvUraUlS5ZAkiXvRNjYjoUXsMEeIBhmjHHCBBwIEAgQGCfhzDExyQmDcwIME+ZMBh8gieEYk0DwGC/E9piIjGNs4w1LtrAlC2F51Wat1tJaen3nj7qS261bre5WV1dX1fM5p0/det+7vG/f6v7WrffeupFSQpKkwQrVboAkaWIyICRJuQwISVIuA0KSlMuAkCTlaqp2A8bKrFmz0qJFi6rdDEmqKcuXL9+WUpqdV1c3AbFo0SKWLVtW7WZIUk2JiBfK1fkRkyQplwEhScplQEiSchkQkqRcBoQkKZcBIUnKZUBIknLVzXUQW/Z08c171o5omYgh6sivHHqZ8dlO2XUNsdDo2jaK7ZSpKrvEmLc5v7IQUIggssdCIXs89FPqV2FAfURQzKsvvPL8YP3h64WmQoGmYtBSLNBULE03Fwo0F4NiIYb8PUoTQd0ExObdB/ja0jXVboY0bM3FGBQipefNxaA5C5WWYtDaVKS1uUBrU+HQ9KTmIpMP/rS88tjWUmRKaxNtLU1MaW2ivbXIlElNTJvUTGtTwVDSiNRNQJz22uk89KV3HVaeyL8h0mjukzTUMqPZTrmqoW7iVH6Z8tspt1C5Ng+1viE3U2ah0bS5bNtG9fssra8/QX9/IiXoT4m+lEgpK0+J/v7SY259f+nx4PO+lLJ5X1lucH1vXz+9fYnuvv7SdH+ip69U3tPXT0//welETzZvT3/2OKC8u7efvV297Njbz4GePg709NPV28f+7j729/TRP8zXckuxwLTJTUyb3EzH5GY62lroaGtmRlsLM9tbOKa9hWOmtDJ7aiuzprQwe2orrU3F4a1cdaluAiICWpocUlFjSakUOvu7+9jX08verl72dffR2dXL3q4+9nb1sudAD7sP9LL7QA+79/eye38PO/d3s3n3Ada8tIcde7vZ39OXu/4Zbc3MnT6ZudNaeU3HZOZ1TGb+jMksmNnGcTPbOKa9xaOSOlY3ASE1ooigpSloaSowneZRr+dATx/bOrvY3tnNts4utu7pYsueLrbsOcCmnQfYtOsAK9bt5OV9Pa9abmprE4tmtXPi7HZOmD2FU+ZM4ZQ5U1l4TDvFgsFR6wwISUxqLjJ/RhvzZ7QNOd/erl427NzPuh37eGH7Pp7fvpfntu3l0edf5scrNg5YX4HXzZ3G4tdO4w3zpvOG+R2cOneqoVFjDAhJw9be2sQpc6Zyypyph9Xt6+7l6c2drNm8h9WbdvPUxt3cvmIj//zIiwC0tRQ5c0EHb1w0k3NPmMlvHjeDSc2OcUxkBoSkMdHW0sQZCzo4Y0HHobL+/sQzWztZsW4nj724k2XP7+Drdz8Nd0NrU4E3LprJOcfP5J2vn8upcw8PHVVXDHXGTC1ZsmRJ8n4Q0sS3vbOLR57bwf1Pb+Nna7awadcBAI6d2so7Fs/h8rPmsWTRzCq3snFExPKU0pLcOgNCUjWt2riLu57cxNJVm1m7pRMonT317tNfwxXnHMdp86ZXuYX1zYCQVBPWv7yPG3+xjlsfW8/GAUcWHz5/IZ94ywlel1EBBoSkmvPg2m1862fP8PO12w6Vvfv0uVzzntdz7NRJVWxZfTEgJNWszq5e/tdPf831Dzx3qOziU2fz5ctP57Udk6vYsvpgQEiqef39ie899Dx/dcdTh8re/hvH8qX3ns7c6R5RjJYBIalu9Pb188NH1/GFH688VPaHFxzPZ99xClNaPXN/pIYKCL+8SFJNaSoW+P3zFrLmS5fwuXeeCsB3fv4cp31xKXev3lzl1tUXA0JSTWptKvJfLj6Jh//8bZx7fOm6iY9/bxmf/v5yevv6q9y6+mBASKppc6dP4v986nz+8WPn0FwMfrLyJU6/5qc8MODsJ42OASGpLrzllNk88hdv54PnHsf+nj4++J1H+PNbn6h2s2qaIzqS6sbM9ha+fPnpnDJnKt9/+AVueWwD61/ez9WXvM4rskfBgJBUdz7ypkWcOHsK1/7709z/9DZmtj/Le8+ax1tPnk3BrxwfNj9iklSXLjh5Fjd96nzmTpvEv6zYyB9891F+8fyOajerphgQkuraT/70Qr794dJp/v9z6Rr+4rYnOVDmFqt6NQNCUl2b0d7ChSfP4pxFM1n/8n7++ZEXWb1pd7WbVRMMCEl1b1JzkZv+6Hy+8XtnAfDNe57hmttX8dy2vVVu2cRmQEhqGMfPamfhMW089uLL3PDg89z22PpqN2lC8ywmSQ3jmCmt3Pu5iwFY/N/+lX3djkUMxYCQ1JDaWorc+Og6lj71Eh2TW/jBJ85l2qTmajdrQvEjJkkN6bNvP4XfWjyHeR2TeXLDLtbt2FftJk04HkFIakgfOm8hHzpvIfes2cLDz+6gq9cv+BvMgJDU0CZl97l+6JntbO/spr2lyHknHOMV1xgQkhrcrCktAHxt6ZpDZbd8+k2cvXBGtZo0YRgQkhrayXOm8rOrLqKzq5fVm3bzuZufYM+Bnmo3a0IwICQ1vEWz2gHoz27B3NtXH7diPlqexSRJmaZC6V9ib78D1uARhCQd0tJUGpjetb+HXftLHzM1FYL21sb8V9mYvZakHJOaS2c0XX3Lk1x9y5MARMA/fexcLjh5VjWbVhUGhCRl5nVM5usfOJNtnd1A6Uji2rufZsPOxryIzoCQpExEcNmZ8w49f2nXAa69+2n6GnRIwkFqSSojG7OmLzXmWU0GhCSVUYzSoHV/vwEhSRrgldNeDQhJ0gAHP2LyCEKS9CrF7Av7GnUMwrOYJKmMgwHx4DPbGZwR5xw/g7MXzqxCq8aPASFJZTQXCsyfMZn7fr2V+3699VV1Z8yfzr9ceUGVWjY+DAhJKqNQCO793MX0DLoQ4jM/eIxNuw5UqVXjx4CQpCEUC0GxUDysLDXAuISD1JI0Qo1yszkDQpJGKIhD946oZwaEJI1QocBhZzXVIwNCkkbIIwhJUr6A+o8HA0KSRqwQ4UdMkqTDBXiaqyTpcAU/YpIk5YlwkFqSlCPC01wlSTkCB6klSTkK4SC1JClHBDTCTeYMCEkaoSBIDXAekwEhSSPkdzFJksqIhviIyRsGSdIIFQK27+3iTf/97iHnKxaDr/72G3jzSbPGqWVjy4CQpBH63SUL6OnrH/Jjpr7+xK2Pb+DJDbsMCElqFGcs6OCMBR1DzrO/u49bH98wTi2qDMcgJEm5DAhJUi4DQpIqqJZPhzUgJEm5DAhJqoCIarfg6BkQkqRcBoQkKZcBIUkVVMtf6mdASJJyGRCSpFwGhCQplwEhScplQEhSBXkltSTpVbxQTpJUtwwISVIuA0KSlMuAkCTlMiAkqQKC2h+lNiAkSbkMCElSLgNCkioo1fCVcgaEJFWAF8pJkuqWASFJymVASJJyGRCSVEE1PEZtQEhSJdTBGLUBIUnKZ0BIknIZEJKkXAaEJFVAZFfK1fAYtQEhScpnQEiSchkQkqRcBoQkVZAXykmSXsUL5SRJdcuAkCTlMiAkSbkmZEBExEURcX9E/H1EXFTt9kjSSB28o1yq4Uvlxi0gIuL6iNgSESsHlV8SEWsiYm1EfD4rTkAnMAlYP15tlCS9YjyPIG4ALhlYEBFF4JvAu4DFwBURsRi4P6X0LuBq4K/GsY2SpMy4BURK6T5gx6Dic4C1KaVnU0rdwI3AZSml/qz+ZaC13Doj4pMRsSwilm3durUi7ZakRlXtMYh5wLoBz9cD8yLityPiH4B/Ar5RbuGU0nUppSUppSWzZ8+ucFMlqbE0VXn7edeSpJTSrcCt490YSRorh77NtXbHqKt+BLEeWDDg+XxgY5XaIkkaoNoB8ShwckQcHxEtwAeA26vcJkkS43ua6w+Bh4BTI2J9RHw8pdQLXAksBVYDN6WUVo1XmyRJ5Y3bGERK6Yoy5XcBd41XOyRJw1Ptj5gkqa7V8Bi1ASFJymdASJJyGRCSpFwGhCRVSAQ1faWcASFJymVASJJyGRCSpFwGhCQplwEhSRUSeKGcJKkO1XxARMSlEXHdrl27qt0USaorNR8QKaU7UkqfnD59erWbIkl1peYDQpImqoN3latVBoQkVVANX0htQEiS8hkQkqRcBoQkKZcBIUkVUttD1AaEJFVUquFrqQ0ISVIuA0KSlMuAkCTlMiAkqUJq/EJqA0KSKskrqSVJdceAkCTlMiAkqUKixi+VMyAkqYJqeAjCgJAk5TMgJEm5aj4gvCe1JFVGzQeE96SWNGHV9hh17QeEJE1kXignSao7BoQkKZcBIUkVUuNDEAaEJCmfASFJFeQtRyVJdceAkCTlMiAkqUK8o5wkqS4ZEJJUSbU7Rm1ASJLyGRCSpFwGhCRViLcclSSVVcNDEAaEJCmfASFJymVASFKFeKGcJKkuGRCSVEGphu85akBIknLVfEBExKURcd2uXbuq3RRJqis1HxAppTtSSp+cPn16tZsiSa9S42PUtR8QkqTKMCAkqYJqeIzagJAk5TMgJKlCosavlDMgJEm5DAhJUi4DQpIqqIbHqA0ISVI+A0KSKqS2h6gNCElSGQaEJFWQF8pJkuqOASFJymVASFKl1PgotQEhScplQEhSBaUavlTOgJAk5TIgJKlCanwIwoCQJOUzICRJuQwISaogr6SuoIhoj4jlEfGfqt0WSWokwwqIiOiIiJsj4lcRsToizh/NxiLi+ojYEhErc+ouiYg1EbE2Ij4/oOpq4KbRbE+SqqlRbjn6deBfU0qvA84AVg+sjIhjI2LqoLKTctZzA3DJ4MKIKALfBN4FLAauiIjFEfF24Clg8zDbKUkaI01HmiEipgFvAT4KkFLqBroHzfZW4NMR8e6U0oGI+ARwOfDugTOllO6LiEU5mzkHWJtSejbb5o3AZcAUoJ1SaOyPiLtSSv3D7p0kadSOGBDACcBW4LsRcQawHPjTlNLegzOklH4UEccDN0bEj4CPAe8YQTvmAesGPF8PnJtSuhIgIj4KbMsLh4i4FLj0pJPyDlgkqbpue3wDD6zdVtFtXPkfTuKyM+eN+XqHExBNwG8Cf5xSeiQivg58HvjCwJlSSn+TvfP/O+DElFLnCNqR90HdobH/lNIN5RZMKd0B3LFkyZJPjGB7klRxn7noRH65fmfFt9PR1lKR9Q4nINYD61NKj2TPb6YUEK8SERcCpwG3AV8ErhxBO9YDCwY8nw9sHMHykjThfOqtJ1a7CUfliIPUKaWXgHURcWpW9DZKA8eHRMRZwLcpjRv8ATAzIr40gnY8CpwcEcdHRAvwAeD2ESwvSRpjwz2L6Y+BH0TEE8CZwFcG1bcB/zml9Ew2TvAR4IXBK4mIHwIPAadGxPqI+DhASqmX0hHHUkpnSN2UUlo1mg5JksZGpFq+zG+AJUuWpGXLllW7GZJUUyJieUppSV7dhL+SWpJUHQaEJCmXASFJymVASJJyGRCSpFx1cxZTRGzllVNrpwO7cqbL1c0CxuJa+MHbGs185eqG6sfg5+Wmx6Kfw+3jkebNqxtO2UTr51jsy8FlvmZ9zY7GaPflwpTS7Nw5U0p19wNclzddrg5YNtbbHe185eqG6sdQfR40fdT9HG4fR9PP4ZRNtH6Oxb4cqi/l6nzN+pqtxL4c/FOvHzHdUWb6SHVjud3Rzleubqh+DH4+Efp4pHnz6oZTNtH6ORb7cnCZr9mx5Wt2dPPVz0dMRyMilqUyF4rUE/tZPxqhj2A/q61ejyBG6rpqN2Cc2M/60Qh9BPtZVR5BSJJyeQQhScplQEiScjVkQERER0R8ptrt0MjVw74bbh8iojN7vCgi7sypf09EHHbzromoHvbbeIqIP4qIDw9Rf01EXFXpdjRkQAAdgC/W2lQP+25M+pBSuj2l9NUxaM94qIf9NmaipOz/35TS36eU/nE825SnUQPiq8CJEbEiIr6W/ayMiCcj4v3VbtxYiIj/mvVpZUR8NiIWRcTqiPh2RKyKiJ9GxORqt3MUBu6770bEewAi4raIuD6b/vjBOxoO/j1Usd0DDezD30bE3RHxWPb6u2yoBSPijRHxeEScEBEfjYhvZOU3RMS1EfFgRDwbEe/LygsR8a1sn98ZEXcdrBtnR+xz9hr9VUR8LyKeiIibI6Itqzs7Iu6NiOURsTQiXpOV/0lEPJXNf+N4dyoi/sfAI6Psnf0Xh+jf6oj4FvAYsCAiOiPiyxHxy4h4OCLmDFjPVUfo4+KI+Fm2v/+kIh082qv3avEHWASszKZ/B/g3oAjMAV4EXlPtNh5l/84GngTagSnAKuAsoBc4M5vnJuBD1W7rUe67DwBfy6Z/ATycTX8XeGe538ME60MTMC2bngWs5ZWzCzuzx4uAO4E3AcuB47LyjwLfyKZvAH5E6U3fYmBtVv4+4K6sfC7wMvC+idjnbJ4EvDmrux64CmgGHgRmZ+XvB67PpjcCrdl0RxX6dRZw74DnTwHHDdG/fuC8AfMn4NJs+m+Av8ymrwGuKtfHrP5BoDXbxnageaz716hHEANdAPwwpdSXUtoM3Au8scptOloXALellPamlDqBW4ELgedSSiuyeZZTesHWsvuBCyNiMaU/zM3ZO8vzKf3xlPs9TCQBfCVKt/P9f8A8Sm9UBvsNSufKX5pSerHMun6cUupPKT01YB0XAD/Kyl8C7hnb5o/KUH1el1J6IJv+PqX2nwqcBvxbRKwA/hKYn83zBKXbIX+I0hugcZVSehw4NiJeGxFnUArgTZTv3wsppYcHrKKbUvhD+b/Jcn38vymlrpTSNmAL+a+bo9I01iusQVHtBlRAuT51DZjuA2rxI6ZDUkobImIGcAlwHzAT+F1K77z3REQt7NsPArOBs1NKPRHxPDApZ75NWflZlN5R5hm4f2PQ40QyVJ8HX5iVKPVhVUrp/Jx1/UfgLcB7gC9ExOtT6R734+lmSkdqc4EbGbp/ewct25OyQwJKf5N5/5MP62NWPvjvecz/nzfqEcQeYGo2fR/w/ogoRsRsSjviF1Vr2di4D3hvRLRFRDtwOaV32/Vg4L4DeAj4LKU+30/pI4mDfZ2ov4eBfZgObMn+kVwMLCyzzE5K/yi+EhEXjWBbPwd+JxuLmEPp46pqGG6fj4uIg0FwBaX2rwFmHyyPiOaIeH2UBnkXpJTuAf6M0kD4lHHoy2A3Uvq4832UwmK4+/SIqt3HhjyCSCltj4gHImIl8BNKh3C/pPRu5c+yQ/GalVJ6LCJu4JWg+w6lQ9+al7Pv7gd+K6W0NiJeoHQUcX8272G/h+wjgaoa1IdHgddFxDJgBfCrIZbbHBGXAj+JiI8Nc3O3AG8DVgK/Bh5h+F99PWZG0OfVwEci4h+Ap4G/Syl1ZwPr10bEdEr/t/43pf58PysL4G9TSjvHsVsApJRWRcRUYENKaVNE/AC4Yzj7dBiK5PRxvA6O/aoNqc5FxJSUUmdEHEMpLN88Ed8ERcQi4M6U0mlVbooyDXkEITWYOyOiA2gB/noihoMmJo8gJEm5GnWQWpJ0BAaEJCmXASFJymVASJJyGRCSpFz/H7IWPN1Vu4UrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "pd.Series([j for k in df[\"clean_text\"].str.split().values for j in k]).value_counts().rank().plot(logy=True, logx=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 2.93 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "8372"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "stopwords=['of', \"the\", \"to\", \"a\"]\n",
    "cv = CountVectorizer(min_df=0.005, # min percentage of documents a token needs to appear in \n",
    "                     max_df=0.6, # max percentage of docs a token is allowed to appear in\n",
    "                     stop_words=stopwords) \n",
    "\n",
    "%time cv.fit(df[\"clean_text\"].values)\n",
    "len(cv.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get the vocabulary from the cv object and filter all words not in the vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Index</th>\n",
       "      <th>date</th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "      <th>week</th>\n",
       "      <th>clean_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>621031</td>\n",
       "      <td>2017-10-26 15:00:54</td>\n",
       "      <td>eBay launches visual search tools that let you shop using photos from your phone or web0eBay today is launching two new visual search tools that will allow online shoppers to use photos they snap,...</td>\n",
       "      <td>eBay launches visual search tools that let you shop using photos from your phone or web</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>ebay launches visual search tools let shop using photos your phone today launching two visual search tools allow online shoppers use photos snap saved phone even those find while browsing web soci...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>621032</td>\n",
       "      <td>2017-10-26 00:00:00</td>\n",
       "      <td>Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller : The SaltScientists are accusing the seed-and-pesticide giant Monsanto of denying the risks of its latest weedkilling...</td>\n",
       "      <td>Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>attacks scientists after studies show trouble accusing giant denying risks latest responded attacking some scientific attacks scientists after studies show trouble scott expert university research...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>621035</td>\n",
       "      <td>2017-10-26 14:41:12</td>\n",
       "      <td>Hard Drive Stats for Q3 2017In Q3 2017, Backblaze introduced both 10 TB and 12 TB hard drives into our data centers, we continued to retire 3 TB and 4 TB hard drives to increase storage density, a...</td>\n",
       "      <td>2017 Hard Drive Reliability by Drive Size and Brand</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>hard drive stats introduced both tb tb hard drives into our data centers continued tb tb hard drives increase storage density added over data storage bring our total storage capacity update well r...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>621036</td>\n",
       "      <td>2017-10-26 15:27:11</td>\n",
       "      <td>This is part 1 of the “How Unreal Renders a Frame” series, you can access part 2 and part 3 as well.I was looking around the Unreal source the other day and inspired by some excellent breakdowns o...</td>\n",
       "      <td>How Unreal Renders a Frame</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>part how renders frame series access part part looking around source day inspired some excellent how popular games render frame thought try something similar well study how renders frame default a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>621423</td>\n",
       "      <td>2017-10-26 16:02:00</td>\n",
       "      <td>Boeing builds towards its first space taxi testWhen NASA first named Boeing and SpaceX as recipients of the Commercial Crew Program, they originally aimed to end the agency's dependence on Russia ...</td>\n",
       "      <td>Boeing builds towards its first space taxi test</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>builds towards first space taxi nasa first named spacex recipients commercial crew program originally aimed end agencys russia developing space technologies hard both companies had push first test...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Index                date  \\\n",
       "0  621031 2017-10-26 15:00:54   \n",
       "1  621032 2017-10-26 00:00:00   \n",
       "2  621035 2017-10-26 14:41:12   \n",
       "3  621036 2017-10-26 15:27:11   \n",
       "4  621423 2017-10-26 16:02:00   \n",
       "\n",
       "                                                                                                                                                                                                      text  \\\n",
       "0  eBay launches visual search tools that let you shop using photos from your phone or web0eBay today is launching two new visual search tools that will allow online shoppers to use photos they snap,...   \n",
       "1  Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller : The SaltScientists are accusing the seed-and-pesticide giant Monsanto of denying the risks of its latest weedkilling...   \n",
       "2  Hard Drive Stats for Q3 2017In Q3 2017, Backblaze introduced both 10 TB and 12 TB hard drives into our data centers, we continued to retire 3 TB and 4 TB hard drives to increase storage density, a...   \n",
       "3  This is part 1 of the “How Unreal Renders a Frame” series, you can access part 2 and part 3 as well.I was looking around the Unreal source the other day and inspired by some excellent breakdowns o...   \n",
       "4  Boeing builds towards its first space taxi testWhen NASA first named Boeing and SpaceX as recipients of the Commercial Crew Program, they originally aimed to end the agency's dependence on Russia ...   \n",
       "\n",
       "                                                                                     title  \\\n",
       "0  eBay launches visual search tools that let you shop using photos from your phone or web   \n",
       "1            Monsanto Attacks Scientists After Studies Show Trouble For Its New Weedkiller   \n",
       "2                                      2017 Hard Drive Reliability by Drive Size and Brand   \n",
       "3                                                               How Unreal Renders a Frame   \n",
       "4                                          Boeing builds towards its first space taxi test   \n",
       "\n",
       "        week  \\\n",
       "0 2017-10-29   \n",
       "1 2017-10-29   \n",
       "2 2017-10-29   \n",
       "3 2017-10-29   \n",
       "4 2017-10-29   \n",
       "\n",
       "                                                                                                                                                                                                clean_text  \n",
       "0  ebay launches visual search tools let shop using photos your phone today launching two visual search tools allow online shoppers use photos snap saved phone even those find while browsing web soci...  \n",
       "1  attacks scientists after studies show trouble accusing giant denying risks latest responded attacking some scientific attacks scientists after studies show trouble scott expert university research...  \n",
       "2  hard drive stats introduced both tb tb hard drives into our data centers continued tb tb hard drives increase storage density added over data storage bring our total storage capacity update well r...  \n",
       "3  part how renders frame series access part part looking around source day inspired some excellent how popular games render frame thought try something similar well study how renders frame default a...  \n",
       "4  builds towards first space taxi nasa first named spacex recipients commercial crew program originally aimed end agencys russia developing space technologies hard both companies had push first test...  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = frozenset(cv.vocabulary_) # for speed\n",
    "\n",
    "# split text into list of tokens then\n",
    "# filter words not in the vocab then\n",
    "# join words\n",
    "df[\"clean_text\"] = df[\"clean_text\"].str.split().apply(lambda wordlist : [word for word in wordlist if word in vocab]).apply(\" \".join)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save Data for next steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"data/sample_5000_clean.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "- 1. Loaded Data\n",
    "- 2. Got to know Data\n",
    "    - 2.1 Distributions\n",
    "    - 2.2 frequent words and stuff\n",
    "- 3. Clean Data\n",
    "    - 3.1 remove weird characters\n",
    "    - 3.2 stopwords\n",
    "    - 3.3 popularity based prefiltering\n",
    "    - 3.4 lemmatization / stemming\n",
    "- 4. Tokenize Data\n",
    "- 5. Save preprocessed data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[1;31mType:\u001b[0m        CountVectorizer\n",
       "\u001b[1;31mString form:\u001b[0m\n",
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "           dtype=<clas <...>               token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b', tokenizer=None,\n",
       "           vocabulary=None)\n",
       "\u001b[1;31mFile:\u001b[0m        c:\\program files (x86)\\anaconda3\\envs\\py37\\lib\\site-packages\\sklearn\\feature_extraction\\text.py\n",
       "\u001b[1;31mDocstring:\u001b[0m  \n",
       "Convert a collection of text documents to a matrix of token counts\n",
       "\n",
       "This implementation produces a sparse representation of the counts using\n",
       "scipy.sparse.csr_matrix.\n",
       "\n",
       "If you do not provide an a-priori dictionary and you do not use an analyzer\n",
       "that does some kind of feature selection then the number of features will\n",
       "be equal to the vocabulary size found by analyzing the data.\n",
       "\n",
       "Read more in the :ref:`User Guide <text_feature_extraction>`.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "input : string {'filename', 'file', 'content'}\n",
       "    If 'filename', the sequence passed as an argument to fit is\n",
       "    expected to be a list of filenames that need reading to fetch\n",
       "    the raw content to analyze.\n",
       "\n",
       "    If 'file', the sequence items must have a 'read' method (file-like\n",
       "    object) that is called to fetch the bytes in memory.\n",
       "\n",
       "    Otherwise the input is expected to be a sequence of items that\n",
       "    can be of type string or byte.\n",
       "\n",
       "encoding : string, 'utf-8' by default.\n",
       "    If bytes or files are given to analyze, this encoding is used to\n",
       "    decode.\n",
       "\n",
       "decode_error : {'strict', 'ignore', 'replace'}\n",
       "    Instruction on what to do if a byte sequence is given to analyze that\n",
       "    contains characters not of the given `encoding`. By default, it is\n",
       "    'strict', meaning that a UnicodeDecodeError will be raised. Other\n",
       "    values are 'ignore' and 'replace'.\n",
       "\n",
       "strip_accents : {'ascii', 'unicode', None}\n",
       "    Remove accents and perform other character normalization\n",
       "    during the preprocessing step.\n",
       "    'ascii' is a fast method that only works on characters that have\n",
       "    an direct ASCII mapping.\n",
       "    'unicode' is a slightly slower method that works on any characters.\n",
       "    None (default) does nothing.\n",
       "\n",
       "    Both 'ascii' and 'unicode' use NFKD normalization from\n",
       "    :func:`unicodedata.normalize`.\n",
       "\n",
       "lowercase : boolean, True by default\n",
       "    Convert all characters to lowercase before tokenizing.\n",
       "\n",
       "preprocessor : callable or None (default)\n",
       "    Override the preprocessing (string transformation) stage while\n",
       "    preserving the tokenizing and n-grams generation steps.\n",
       "    Only applies if ``analyzer is not callable``.\n",
       "\n",
       "tokenizer : callable or None (default)\n",
       "    Override the string tokenization step while preserving the\n",
       "    preprocessing and n-grams generation steps.\n",
       "    Only applies if ``analyzer == 'word'``.\n",
       "\n",
       "stop_words : string {'english'}, list, or None (default)\n",
       "    If 'english', a built-in stop word list for English is used.\n",
       "    There are several known issues with 'english' and you should\n",
       "    consider an alternative (see :ref:`stop_words`).\n",
       "\n",
       "    If a list, that list is assumed to contain stop words, all of which\n",
       "    will be removed from the resulting tokens.\n",
       "    Only applies if ``analyzer == 'word'``.\n",
       "\n",
       "    If None, no stop words will be used. max_df can be set to a value\n",
       "    in the range [0.7, 1.0) to automatically detect and filter stop\n",
       "    words based on intra corpus document frequency of terms.\n",
       "\n",
       "token_pattern : string\n",
       "    Regular expression denoting what constitutes a \"token\", only used\n",
       "    if ``analyzer == 'word'``. The default regexp select tokens of 2\n",
       "    or more alphanumeric characters (punctuation is completely ignored\n",
       "    and always treated as a token separator).\n",
       "\n",
       "ngram_range : tuple (min_n, max_n), default=(1, 1)\n",
       "    The lower and upper boundary of the range of n-values for different\n",
       "    word n-grams or char n-grams to be extracted. All values of n such\n",
       "    such that min_n <= n <= max_n will be used. For example an\n",
       "    ``ngram_range`` of ``(1, 1)`` means only unigrams, ``(1, 2)`` means\n",
       "    unigrams and bigrams, and ``(2, 2)`` means only bigrams.\n",
       "    Only applies if ``analyzer is not callable``.\n",
       "\n",
       "analyzer : string, {'word', 'char', 'char_wb'} or callable\n",
       "    Whether the feature should be made of word n-gram or character\n",
       "    n-grams.\n",
       "    Option 'char_wb' creates character n-grams only from text inside\n",
       "    word boundaries; n-grams at the edges of words are padded with space.\n",
       "\n",
       "    If a callable is passed it is used to extract the sequence of features\n",
       "    out of the raw, unprocessed input.\n",
       "\n",
       "    .. versionchanged:: 0.21\n",
       "\n",
       "    Since v0.21, if ``input`` is ``filename`` or ``file``, the data is\n",
       "    first read from the file and then passed to the given callable\n",
       "    analyzer.\n",
       "\n",
       "max_df : float in range [0.0, 1.0] or int, default=1.0\n",
       "    When building the vocabulary ignore terms that have a document\n",
       "    frequency strictly higher than the given threshold (corpus-specific\n",
       "    stop words).\n",
       "    If float, the parameter represents a proportion of documents, integer\n",
       "    absolute counts.\n",
       "    This parameter is ignored if vocabulary is not None.\n",
       "\n",
       "min_df : float in range [0.0, 1.0] or int, default=1\n",
       "    When building the vocabulary ignore terms that have a document\n",
       "    frequency strictly lower than the given threshold. This value is also\n",
       "    called cut-off in the literature.\n",
       "    If float, the parameter represents a proportion of documents, integer\n",
       "    absolute counts.\n",
       "    This parameter is ignored if vocabulary is not None.\n",
       "\n",
       "max_features : int or None, default=None\n",
       "    If not None, build a vocabulary that only consider the top\n",
       "    max_features ordered by term frequency across the corpus.\n",
       "\n",
       "    This parameter is ignored if vocabulary is not None.\n",
       "\n",
       "vocabulary : Mapping or iterable, optional\n",
       "    Either a Mapping (e.g., a dict) where keys are terms and values are\n",
       "    indices in the feature matrix, or an iterable over terms. If not\n",
       "    given, a vocabulary is determined from the input documents. Indices\n",
       "    in the mapping should not be repeated and should not have any gap\n",
       "    between 0 and the largest index.\n",
       "\n",
       "binary : boolean, default=False\n",
       "    If True, all non zero counts are set to 1. This is useful for discrete\n",
       "    probabilistic models that model binary events rather than integer\n",
       "    counts.\n",
       "\n",
       "dtype : type, optional\n",
       "    Type of the matrix returned by fit_transform() or transform().\n",
       "\n",
       "Attributes\n",
       "----------\n",
       "vocabulary_ : dict\n",
       "    A mapping of terms to feature indices.\n",
       "\n",
       "fixed_vocabulary_: boolean\n",
       "    True if a fixed vocabulary of term to indices mapping\n",
       "    is provided by the user\n",
       "\n",
       "stop_words_ : set\n",
       "    Terms that were ignored because they either:\n",
       "\n",
       "      - occurred in too many documents (`max_df`)\n",
       "      - occurred in too few documents (`min_df`)\n",
       "      - were cut off by feature selection (`max_features`).\n",
       "\n",
       "    This is only available if no vocabulary was given.\n",
       "\n",
       "Examples\n",
       "--------\n",
       ">>> from sklearn.feature_extraction.text import CountVectorizer\n",
       ">>> corpus = [\n",
       "...     'This is the first document.',\n",
       "...     'This document is the second document.',\n",
       "...     'And this is the third one.',\n",
       "...     'Is this the first document?',\n",
       "... ]\n",
       ">>> vectorizer = CountVectorizer()\n",
       ">>> X = vectorizer.fit_transform(corpus)\n",
       ">>> print(vectorizer.get_feature_names())\n",
       "['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']\n",
       ">>> print(X.toarray())\n",
       "[[0 1 1 1 0 0 1 0 1]\n",
       " [0 2 0 1 0 1 1 0 1]\n",
       " [1 0 0 1 1 0 1 1 1]\n",
       " [0 1 1 1 0 0 1 0 1]]\n",
       ">>> vectorizer2 = CountVectorizer(analyzer='word', ngram_range=(2, 2))\n",
       ">>> X2 = vectorizer2.fit_transform(corpus)\n",
       ">>> print(vectorizer2.get_feature_names())\n",
       "['and this', 'document is', 'first document', 'is the', 'is this',\n",
       "'second document', 'the first', 'the second', 'the third', 'third one',\n",
       " 'this document', 'this is', 'this the']\n",
       " >>> print(X2.toarray())\n",
       " [[0 0 1 1 0 0 1 0 0 0 0 1 0]\n",
       " [0 1 0 1 0 1 0 1 0 0 1 0 0]\n",
       " [1 0 0 1 0 0 0 0 1 1 0 1 0]\n",
       " [0 0 1 0 1 0 1 0 0 0 0 0 1]]\n",
       "\n",
       "See Also\n",
       "--------\n",
       "HashingVectorizer, TfidfVectorizer\n",
       "\n",
       "Notes\n",
       "-----\n",
       "The ``stop_words_`` attribute can get large and increase the model size\n",
       "when pickling. This attribute is provided only for introspection and can\n",
       "be safely removed using delattr or set to None before pickling.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?cv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py37",
   "language": "python",
   "name": "py37"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}