Doc2vec for Investigative Journalism.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Doc2vec for Investigative Journalism\n",
    "\n",
    "\n",
    "[Read our article about using doc2vec for the Mauritius Leaks](https://qz.com/1670632/how-quartz-used-ai-to-help-reporters-search-the-mauritius-leaks/)\n",
    "\n",
    "Doc2vec is a powerful tool we've used to help search the Mauritius Leaks documents. But it's generally useful, creating a multipurpose, generic notion of **similarity** that you can use to find documents about the same thing as documents you already have.\n",
    "\n",
    "\n",
    "### Goal: Find docs about _homelessness_\n",
    "\n",
    "#### That don't contain the word \"homelessness\".\n",
    "\n",
    "In this tutorial, we're going to look at some emails from the office of New York City mayor Bill de Blasio that were released under the Freedom of Information Law. (The emails were part of the \"Agent of the City\" hubbub; you can download the original file [here](https://a860-openrecords.nyc.gov/response/120252?token=c784372fd140497081b4bfcff9f0e3a0).)\n",
    "\n",
    "Let's **pretend** that we're reporters, that we're interested in writing about what the de Blasio administration has said over time, interally, about its plan to address **homelessness** in the city. And let's pretend that we have a keyword search system for these emails; for these, the search bar in our PDF viewer would work fine, but for other, larger leaks with heterogenous filetypes (like the Mauritius Leaks), a more intense software solution would be necessary. \n",
    "\n",
    "Since we have a keyword search system, we can just search for \"homelessness\" and get a lot of results. But what if we want to find the ones we're missing, the ones that don't contain that keyword?\n",
    "\n",
    "\n",
    "### Dependencies\n",
    "You'll need to have the following dependencies installed to run this notebook.\n",
    "\n",
    "* `pip install pypdf2 gensim scipy`\n",
    "* `brew install elasticsearch` (or `apt-get install elasticsearch`... and I'm sure there's a way to do it on Windows.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 0. Convert our documents to JSONL\n",
    "\n",
    "As part of the training process, we'll have to iterate over our documents several times. Rather than re-parse the PDF each time, we're going to just parse it once and write the results to a file. JSONL is the format I've chosen to store the text of the document pages in; the format (`{\"_source\": {\"content\": \"the actual content }}`) is meant to mimic the output of Apache Tika. JSONL is just a plain text file with one JSON object per line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import PyPDF2\n",
    "import json\n",
    "from os.path import exists\n",
    "jsonl_file = \"nyc_docs.jsonl\"\n",
    "if not exists(jsonl_file):\n",
    "    pdf_file = open('2018.05.24_BerlinRosen_Responsive_Records.pdf', 'rb')\n",
    "    read_pdf = PyPDF2.PdfFileReader(pdf_file)\n",
    "    with open(jsonl_file, 'w') as f:\n",
    "        for page_num in range(read_pdf.getNumPages()):\n",
    "            page = read_pdf.getPage(page_num)\n",
    "            page_content = page.extractText().encode('utf-8').decode(\"utf-8\") \n",
    "            f.write(json.dumps({\"_source\": {\"content\": page_content}, \"_id\": f\"p{page_num}\"}) + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1. Train a model with Doc2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the resulting model is in the repo. (it takes about 2 minutes and 30 seconds to train)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utilities import *\n",
    "from os import makedirs\n",
    "import logging\n",
    "from tqdm import tqdm_notebook\n",
    "from datetime import datetime\n",
    "start = datetime.now()\n",
    "\n",
    "model_filename = \"models/nyc_docs.jsonl.model\"\n",
    "\n",
    "if not exists(model_filename):\n",
    "    # these are some settings you might want to change, under certain circumstances.\n",
    "    num_epochs = 20\n",
    "    vector_size = 100\n",
    "    window = 5\n",
    "    alpha = 0.025 # aka learning rate\n",
    "    min_count = 5\n",
    "\n",
    "    model = Doc2Vec(\n",
    "        vector_size=vector_size, \n",
    "        dbow_words = 1, \n",
    "        dm=0,\n",
    "        epochs=1, \n",
    "        workers=4,\n",
    "        window=window, \n",
    "        seed=1337, \n",
    "        min_count=min_count,\n",
    "        alpha=alpha, \n",
    "        min_alpha=alpha\n",
    "      )\n",
    "    makedirs(dirname(model_filename), exist_ok=True)\n",
    "\n",
    "    vocab_iterator = SugarcaneJsonlIterator(jsonl_file, ngrams_type=\"bigrams\")\n",
    "    model.build_vocab(vocab_iterator)\n",
    "\n",
    "    for epoch in tqdm_notebook(range(num_epochs)):\n",
    "        model.train(SugarcaneJsonlIterator(jsonl_file, ngrams_type=\"bigrams\"),\n",
    "            total_examples=model.corpus_count, \n",
    "            epochs=1\n",
    "          )\n",
    "        model.save(model_filename)\n",
    "\n",
    "    print(\"finish training w2v\" +  str(datetime.now()))\n",
    "    print(\"training w2v took {} seconds ({}h {}m)\".format(int((datetime.now() - start).seconds), (datetime.now() - start).seconds // 3600, ((datetime.now() - start).seconds % 3600) // 60))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## step 2: put documents into ElasticSearch\n",
    "\n",
    "We need a way to search the documents (to simulate what a reporter with a keyword-search tool could do) and a way to query individual documents whose IDs are returned by the doc2vec model. I chose to do this with ElasticSearch, but Postgresql or other tools would work fine. \n",
    "\n",
    "This assumes ElasticSearch is running locally on port 9200. (If you installed ElasticSearch with `brew`, you may have to run `brew services start elasticsearch` to actually start ElasticSearch.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "index_name = \"nycdocs\"\n",
    "\n",
    "try:\n",
    "    requests.put(f\"http://localhost:9200/{index_name}\", {})\n",
    "except:\n",
    "    pass # it's okay if we get an error because the index already exists.\n",
    "\n",
    "with open(jsonl_file, 'r') as f:\n",
    "    for line in f:\n",
    "        line = json.loads(line)\n",
    "        source = line[\"_source\"]\n",
    "        source[\"id\"] = line[\"_id\"]\n",
    "        requests.post(f\"http://localhost:9200/{index_name}/_doc/\", data=json.dumps(source), headers={\"Content-Type\": \"application/json\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's make sure that there's data in the ElasticSearch index before we go forward.\n",
    "ret = requests.get(f\"http://localhost:9200/{index_name}/_count\")\n",
    "assert json.loads(ret.content)[\"count\"] > 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## step 3: Querying our Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import Doc2Vec\n",
    "model = Doc2Vec.load(model_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.14892638,  0.84225196,  0.5657305 , -0.05731661,  1.0092884 ,\n",
       "       -0.16890779, -0.9475695 , -0.31454888,  0.42846683,  0.9906296 ,\n",
       "        1.3255194 , -0.9393652 ,  1.0227212 ,  0.05691548,  0.288966  ,\n",
       "        0.4966944 , -0.3193189 ,  0.52391315,  0.23011431, -0.55094516,\n",
       "        0.5415337 ,  0.53747773, -0.29408413, -0.4506765 ,  0.58461386,\n",
       "       -1.0946444 ,  0.58335054, -0.16434972, -0.24865314, -0.15691808,\n",
       "       -0.08979879, -0.23274064, -0.55003446,  0.8960646 ,  0.24233627,\n",
       "       -0.45479947, -0.1096103 , -0.22644855, -0.04939383, -0.65178746,\n",
       "       -0.32949883,  0.54339314, -0.9402834 , -0.10056758,  0.27990496,\n",
       "       -0.52813953, -0.38317814,  0.54720885, -0.05945592,  0.9192372 ,\n",
       "        0.36784324, -0.09500701, -0.08845058,  1.002136  , -0.80019456,\n",
       "        0.3984145 ,  0.21191886, -0.36903706,  0.03561793,  0.72101   ,\n",
       "       -0.03929467,  0.22953278, -0.21876392,  0.7107592 ,  0.28367153,\n",
       "       -0.7346413 , -0.26492935,  0.7894046 , -0.06782962, -0.49392605,\n",
       "        0.4354293 , -0.5209322 ,  1.0260226 , -0.8264296 ,  0.58497036,\n",
       "       -0.26572046, -0.4190991 , -0.3194183 , -0.37581515, -0.7678954 ,\n",
       "       -0.20899788,  0.04178713,  0.21122219, -0.4126727 ,  0.35515517,\n",
       "       -0.06052636,  0.31234804,  0.11329623,  0.18822236,  0.5406627 ,\n",
       "        0.04603505, -0.23385243, -0.23653832,  1.0472375 , -0.46508786,\n",
       "        0.12786631, -0.28991586,  0.3373969 , -0.41665322, -0.06130537],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "from elasticsearch import Elasticsearch\n",
    "from elasticsearch_dsl import Search\n",
    "import csv \n",
    "class Doc2vecSearcher:\n",
    "    def __init__(self, client, d2v_model):\n",
    "        self.client = client\n",
    "        self.model = d2v_model\n",
    "        \n",
    "    def has_term_in_content(self, term, es_id):\n",
    "        r = Search(using=client, index=index_name) \\\n",
    "        .query(\"match\", id=es_id)\n",
    "        resp = r.execute()\n",
    "        return term.lower() in resp[0].content.lower()\n",
    "\n",
    "    def search_for_documents_like_this_that_search_would_miss(self, documents, search_term, topn=300):\n",
    "        \"\"\"find documents similar to those found by a given search, but which are missing the search term\"\"\"\n",
    "        maybe_matches = self.model.docvecs.most_similar(documents, topn=topn)\n",
    "\n",
    "        # filter out filepaths (since those are useless to me.)\n",
    "        maybe_matches = [id_score for id_score in maybe_matches if '/' not in id_score[0]]\n",
    "\n",
    "        # add metadata to rows.\n",
    "        output = []\n",
    "        for match in maybe_matches:\n",
    "            row = {\n",
    "                \"page_num\": match[0],\n",
    "                \"url\": \"http://localhost:9200/nycdocs/_search?q=id:\" + match[0],\n",
    "                \"score\": match[1],\n",
    "                \"in_documents\": match[0] in documents,\n",
    "                \"matches_search_term\": self.has_term_in_content(search_term, match[0]),\n",
    "            }\n",
    "            output.append(row)\n",
    "        return output\n",
    "\n",
    "    def to_csv(self, search_results, search_term=None, csv_fn=None):\n",
    "        search_term_cln = search_term.replace(\" \", \"_\")\n",
    "        with open(csv_fn if csv_fn else f\"{search_term_cln}.csv\", 'w', newline='') as csvfile:\n",
    "            fieldnames = [\"page_num\", \n",
    "                          \"url\", \n",
    "                          \"score\", \n",
    "                          f\"matches search term (\\\"{search_term}\\\")\" if search_term else \"matches search term\",\n",
    "                         ]\n",
    "            writer = csv.DictWriter(csvfile, fieldnames=fieldnames, extrasaction='ignore')\n",
    "\n",
    "            writer.writeheader()\n",
    "            for row in search_results:\n",
    "                row = row.copy()\n",
    "                row[f\"matches search term (\\\"{search_term}\\\")\" if search_term else \"matches search term\"] = row[\"matches_search_term\"]\n",
    "                writer.writerow(row)\n",
    "client = Elasticsearch()\n",
    "doc_searcher = Doc2vecSearcher(client, model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### finding emails about homelessness\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "here are 5 documents that the model thinks are most similar to our examples\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'page_num': 'p1972',\n",
       "  'url': 'http://localhost:9200/nycdocs/_search?q=id:p1972',\n",
       "  'score': 0.5875276327133179,\n",
       "  'in_documents': False,\n",
       "  'matches_search_term': False},\n",
       " {'page_num': 'p1693',\n",
       "  'url': 'http://localhost:9200/nycdocs/_search?q=id:p1693',\n",
       "  'score': 0.5547754168510437,\n",
       "  'in_documents': False,\n",
       "  'matches_search_term': False},\n",
       " {'page_num': 'p1103',\n",
       "  'url': 'http://localhost:9200/nycdocs/_search?q=id:p1103',\n",
       "  'score': 0.5315240621566772,\n",
       "  'in_documents': False,\n",
       "  'matches_search_term': False},\n",
       " {'page_num': 'p1054',\n",
       "  'url': 'http://localhost:9200/nycdocs/_search?q=id:p1054',\n",
       "  'score': 0.5300723910331726,\n",
       "  'in_documents': False,\n",
       "  'matches_search_term': False},\n",
       " {'page_num': 'p1077',\n",
       "  'url': 'http://localhost:9200/nycdocs/_search?q=id:p1077',\n",
       "  'score': 0.5211280584335327,\n",
       "  'in_documents': False,\n",
       "  'matches_search_term': False}]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "known_pages_about_homelessness = [ \n",
    "    \"p1154\",\n",
    "    \"p1262\",\n",
    "    \"p3813\"\n",
    "]\n",
    "\n",
    "other_homelessness_docs = doc_searcher.search_for_documents_like_this_that_search_would_miss(known_pages_about_homelessness, \"homelessness\")\n",
    "matches = [doc for doc in other_homelessness_docs if not doc[\"in_documents\"] and not doc[\"matches_search_term\"]]\n",
    "print(\"here are 5 documents that the model thinks are most similar to our examples\")\n",
    "print(\"BUT which don't contain the word 'homelessness'\")\n",
    "matches[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if we want to, we can write these docs to a CSV, to look at them later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_searcher.to_csv(other_homelessness_docs, \"homelessness\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}