06_Keyword_Searching_and_ErrorAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keyword Searching and Error Analysis\n",
    "\n",
    "A good starting point to determine the complexity of your problem is a keyword search: if a keyword or set of keywords appear in the document, classify the document as positive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#First import packages\n",
    "import urllib.request\n",
    "import os\n",
    "import codecs\n",
    "import zipfile\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "import ipywidgets\n",
    "import sklearn.metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nlp_pneumonia_utils import read_doc_annotations\n",
    "from nlp_pneumonia_utils import calculate_prediction_metrics\n",
    "from nlp_pneumonia_utils import mark_text\n",
    "from nlp_pneumonia_utils import pneumonia_annotation_html_markup\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "import ipywidgets as widgets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Create a keyword classifier\n",
    "### Team exercise:\n",
    "### iterate through a set of keywords (self.keywords)\n",
    "### If a keyword in the set appears in 'text', return prediction = 1\n",
    "### else, return prediction = 0. \n",
    "Complete the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class KeywordClassifier(object):\n",
    "    def __init__(self):\n",
    "        self.keywords = set()\n",
    "    def predict(self, text):\n",
    "        prediction = 0\n",
    "#         your code here\n",
    "        return prediction\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the function you just wrote by adding one keyword to the set: 'pneumonia'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading annotations from file : data/training_v2.zip\n",
      "Opening local file : data/training_v2.zip\n",
      "Total Annotated Documents : 70\n",
      "Number of documents labeled positive by keyword classifier: 30\n",
      "Number of documents labeled positive by gold standard: 34\n"
     ]
    }
   ],
   "source": [
    "keyword_classifier = KeywordClassifier()\n",
    "keyword_classifier.keywords.add('pneumonia')\n",
    "annotated_doc_map = read_doc_annotations('data/training_v2.zip')\n",
    "print('Total Annotated Documents : {0}'.format(len(annotated_doc_map)))\n",
    "\n",
    "keyword_positives = 0\n",
    "actual_positives = 0\n",
    "for doc in annotated_doc_map.values():\n",
    "    if keyword_classifier.predict(doc.text) == 1:\n",
    "        keyword_positives+=1\n",
    "        \n",
    "    if doc.positive_label == 1:\n",
    "        actual_positives+=1\n",
    "        \n",
    "print('Number of documents labeled positive by keyword classifier: ' + str(keyword_positives))\n",
    "print('Number of documents labeled positive by gold standard: ' + str(actual_positives))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Error analysis of keyword classifier\n",
    "\n",
    "### Let's first look at false negatives \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def list_false_negatives(gold_docs, prediction_function):\n",
    "    fn_docs={}\n",
    "    for doc_name, gold_doc in gold_docs.items():\n",
    "        gold_label=gold_doc.positive_label;\n",
    "        pred_label = prediction_function(gold_doc.text)\n",
    "        if gold_label==1 and pred_label==0:\n",
    "            fn_docs[doc_name]=gold_doc            \n",
    "    return fn_docs     \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13\n"
     ]
    }
   ],
   "source": [
    "fn=list_false_negatives(annotated_doc_map, keyword_classifier.predict)\n",
    "docs=list(fn.keys())\n",
    "print(len(docs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How do we fix false negatives?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look through each document that is a false negative, showing  one document a time:<br/><br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef07cb4a927f4b829c5aecd3541c90ca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='i', max=12), Output()), _dom_classes=('widget-interact',…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact(i=ipywidgets.IntSlider(min=0, max=len(docs)-1))\n",
    "def display_doc(i):\n",
    "    doc_name=docs[i]    \n",
    "    print(doc_name)\n",
    "    anno_doc=fn[doc_name]\n",
    "    display(HTML(pneumonia_annotation_html_markup(anno_doc).replace('\\n', '<br>')))    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. More efficient review:\n",
    "Not convenient to read? Let's try snippet view instead. Now we need to make another function to replace \"*pneumonia_annotation_html_markup*\". \n",
    "\n",
    "Although we measuring the document level annotation, we will focus on mention level (\"**EVIDENCE_OF_PNEUMONIA**\") error analyses. Because the later is where the errors originate from.<br/><br/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def snippets_markup(annotated_doc_map):\n",
    "    html = [\"<html>\",\"<table width=100% >\",\n",
    "            \"<col style=\\\"width:25%\\\"><col style=\\\"width:75%\\\">\"\n",
    "            \"<tr><th style=\\\"text-align:center\\\">document name</th><th style=\\\"text-align:center\\\">Snippets</th>\"]\n",
    "    for doc_name, anno_doc in annotated_doc_map.items():\n",
    "        html.extend(snippet_markup(doc_name,anno_doc))\n",
    "    html.append(\"</table>\")\n",
    "    html.append(\"</html>\")\n",
    "    return ''.join(html) \n",
    "\n",
    "\n",
    "def snippet_markup(doc_name,anno_doc):\n",
    "    from pyConTextNLP.display.html import __sort_by_span\n",
    "    from pyConTextNLP.display.html import __insert_color\n",
    "    html=[]\n",
    "    color= 'blue'    \n",
    "    window_size=50    \n",
    "    html.append(\"<tr>\")\n",
    "    html.append(\"<td style=\\\"text-align:left\\\">{0}</td>\".format(doc_name))\n",
    "    html.append(\"<td></td>\")\n",
    "    html.append(\"</tr>\")\n",
    "    for anno in anno_doc.annotations:\n",
    "        if anno.type == 'EVIDENCE_OF_PNEUMONIA':\n",
    "#           make sure the our snippet will be cut inside the text boundary\n",
    "            begin=anno.start_index-window_size\n",
    "            end=anno.end_index+window_size\n",
    "            begin=begin if begin>0 else 0\n",
    "            end=end if end<len(anno_doc.text) else len(anno_doc.text)    \n",
    "#           render a highlighted snippet\n",
    "            cell=__insert_color(anno_doc.text[begin:end],[anno.start_index-begin,anno.end_index-end],color)\n",
    "#           add the snippet into table\n",
    "            html.append(\"<tr>\")\n",
    "            html.append(\"<td></td>\")\n",
    "            html.append(\"<td style=\\\"text-align:left\\\">{0}</td>\".format(cell))\n",
    "            html.append(\"</tr>\") \n",
    "    return html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try it out:<br/><br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><table width=100% ><col style=\"width:25%\"><col style=\"width:75%\"><tr><th style=\"text-align:center\">document name</th><th style=\"text-align:center\">Snippets</th><tr><td style=\"text-align:left\">subject_id_6349_hadm_id_20192</td><td></td></tr><tr><td></td><td style=\"text-align:left\">There is no pulmonary vascular\n",
       "     engorgement.  <span style=\"color: blue;\">There is an increasing left-sided pleural effusion with\n",
       "     associated atelectasis.  Pneumonia at this location cannot be excluded.</span>  Noted\n",
       "     is a density in the left upper lung z</td></tr><tr><td style=\"text-align:left\">subject_id_157_hadm_id_26180</td><td></td></tr><tr><td></td><td style=\"text-align:left\">ung is incompletely imaged\n",
       "     on this study and <span style=\"color: blue;\">there is a questionable area of abnormality partially\n",
       "     obscuring the mid portion of the right hemidiaphragm</span>, incompletely evaluated.\n",
       "     \n",
       "     IMPRESSION:  </td></tr><tr><td style=\"text-align:left\">subject_id_150_hadm_id_12121</td><td></td></tr><tr><td></td><td style=\"text-align:left\">es are\n",
       "     unremarkable.\n",
       "     \n",
       "     IMPRESSION:  <span style=\"color: blue;\">Small focal opacity in right upper lobe</span> and right paratracheal\n",
       "     opacity.  In the sett</td></tr><tr><td></td><td style=\"text-align:left\">CHEST PA AND LATERAL:  The heart size is normal.  <span style=\"color: blue;\">There is an area of\n",
       "     increased opacity</span> lateral to the right paratracheal stripe.  In the</td></tr><tr><td></td><td style=\"text-align:left\">pacity lateral to the right paratracheal stripe.  <span style=\"color: blue;\">In the right\n",
       "     upper lobe, there is a small focal opacity.</span>  The lungs are otherwise clear.\n",
       "     There are no</td></tr><tr><td style=\"text-align:left\">subject_id_7675_hadm_id_3</td><td></td></tr><tr><td></td><td style=\"text-align:left\">r contours are stable\n",
       "     since the prior study. <span style=\"color: blue;\">There is persistent opacification in the right lower\n",
       "     lung zone</span>. The pulmonary vascularity is unremarkable. The e</td></tr><tr><td></td><td style=\"text-align:left\">terminating in the left IJ. No\n",
       "     pneumothorax. <span style=\"color: blue;\">Persistent right lower lung zone opacification</span> and probable\n",
       "     small effusion.\n",
       "\n",
       "</td></tr><tr><td style=\"text-align:left\">subject_id_9082_hadm_id_29395</td><td></td></tr><tr><td></td><td style=\"text-align:left\">tient with seizure.\n",
       "     \n",
       "     Low lung volumes.  <span style=\"color: blue;\">Bilateral basilar opacities,</span> considerably larger at the\n",
       "     left base than at</td></tr><tr><td></td><td style=\"text-align:left\">hyroid.\n",
       "     \n",
       "     IMPRESSION:  Lung volumes with <span style=\"color: blue;\">bilateral basilar opacities.</span>\n",
       "     \n",
       "     Question substernal thyroid enlargemen</td></tr><tr><td style=\"text-align:left\">subject_id_5472_hadm_id_11987</td><td></td></tr><tr><td></td><td style=\"text-align:left\">id SVC. \n",
       "     There is no apparent pneumothorax.  <span style=\"color: blue;\">A right IJ line, NGT, and ETT are\n",
       "     unchanged as are the parenchymal changes in the lungs</span> compared to the earlier\n",
       "     chest x-ray this mor</td></tr></table></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fn=list_false_negatives(annotated_doc_map, keyword_classifier.predict)\n",
    "docs=list(fn.keys())\n",
    "display(HTML(snippets_markup(fn)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Add more keywords to your keyword classifier to decrease the number of false negatives\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'pneumonia', 'consolidation'}\n"
     ]
    }
   ],
   "source": [
    "keyword_classifier.keywords = {'pneumonia', 'consolidation'}\n",
    "print(keyword_classifier.keywords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Compute the outcome metrics using your KeywordClassifier\n",
    "\n",
    "Let's copy and paste the *keyword_classifier* definition below for editing convenience. Reuse the *calculate_prediction_metrics* function. We need to change a little bit of its input parameter, because it needs a list for the 1st parameter, and our *annotated_doc_map* is a dictionary. So we use *list(annotated_doc_map.values())* instead of *annotated_doc_map* directly.<br/><br/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision : 0.6829268292682927\n",
      "Recall :    0.8235294117647058\n",
      "F1:         0.7466666666666667\n",
      "\n",
      "Confusion Matrix : \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>23</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted   0   1\n",
       "Actual           \n",
       "0          23  13\n",
       "1           6  28"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "calculate_prediction_metrics(list(annotated_doc_map.values()), keyword_classifier.predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Quiz\n",
    "Try the following questions, see if you've understood this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa405de456344aa3ac6532dd70750283",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "RadioButtons(description='False negative means:', layout=Layout(width='600px'), options=('Negative in both gol…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9caec9fef713456f83e5f5d77c42c59b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Button(description='Submit', style=ButtonStyle())"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from quiz_utils import error_analyses_1\n",
    "error_analyses_1()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><body>What type of data will be returned by the function <b>\"list_false_negatives\"</b>?</body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d39aea50c9342769af995c43a14d289",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "RadioButtons(description='Which is corret:', layout=Layout(width='600px'), options=('list', 'array', 'dictiona…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "909fc3ee048746149285e600f22e6cc3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Button(description='Submit', style=ButtonStyle())"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from quiz_utils import error_analyses_2\n",
    "error_analyses_2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><body>If you see the following expert annotation in gold reference, but your KeywordClassifier predict the corresponding document as negative answer. Which keyword above is the best choice to add in your KeywordClassifier?<br/><br/>The mediastinal and hilar contours are\n",
       "unremarkable. There is <span style=\"color: blue;\">patchy opacity at the left lower lobe representing\n",
       "either atelectasis or consolidation</span>. No definite free air is identified.</body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5a0d8f1daf724392bf7a687d643a4413",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "RadioButtons(description='Choose best keyword:', layout=Layout(width='600px'), options=('is', 'lobe', 'patchy …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b9bc4b1cdcc349a7911085ac889ef568",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Button(description='Submit', style=ButtonStyle())"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from quiz_utils import error_analyses_3\n",
    "error_analyses_3()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/><br/>This material presented as part of the DeCART Data Science for the Health Science Summer Program at the University of Utah in 2019.<br/>\n",
    "Presenters : Dr. Wendy Chapman, Kelly Peterson, Alec Chapman, Jianlin Shi <br> Acknowledgement: Many thanks to Olga Patterson because part of the materials are adopted from his previous work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}