09_NLP_pneumonia_pyConText_targets_and_modifiers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objectives\n",
    "* Gain hands on experience working with pyConText and its resources\n",
    "* Understand and develop Targets\n",
    "* Understand and develop Modifiers\n",
    "* Graph and visualize Targets and Modifiers together\n",
    "* Gain tools for group project in reducing False Negatives and False Positives (F1 measure)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7.0.0\n",
      "Imported pneumonia nlp utilities...\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "# we will definitely need pyConText\n",
    "import pyConTextNLP\n",
    "from pyConTextNLP import pyConText\n",
    "from pyConTextNLP import itemData\n",
    "import urllib\n",
    "from radnlp.data import classrslts \n",
    "from pyConTextNLP.display.html import mark_document_with_html\n",
    "print(pyConTextNLP.__version__)\n",
    "# useful utilities in RadNLP as well\n",
    "\n",
    "import pandas as pd\n",
    "# packages for interaction\n",
    "from ipywidgets import interact, interactive, fixed\n",
    "from IPython.display import display, HTML, Image\n",
    "import ipywidgets\n",
    "# and also our utilities for this class\n",
    "from nlp_pneumonia_utils import Annotation\n",
    "from nlp_pneumonia_utils import AnnotatedDocument\n",
    "from nlp_pneumonia_utils import read_brat_annotations\n",
    "from nlp_pneumonia_utils import read_doc_annotations\n",
    "from nlp_pneumonia_utils import read_annotations\n",
    "from nlp_pneumonia_utils import calculate_prediction_metrics\n",
    "from nlp_pneumonia_utils import mark_text\n",
    "from nlp_pneumonia_utils import clearPyConTextRegularExpressions\n",
    "from nlp_pneumonia_utils import pneumonia_annotation_html_markup\n",
    "from nlp_pneumonia_utils import mark_document_with_html\n",
    "from nlp_pneumonia_utils import view_single_sentence_graph\n",
    "from nlp_pneumonia_utils import markup_sentence\n",
    "\n",
    "from pycontext_quiz import identify_target_category\n",
    "from pycontext_quiz import file_delimiter_quiz\n",
    "from pycontext_quiz import modifier_directionality_quiz\n",
    "from pycontext_quiz import second_most_frequent_modifier_quiz\n",
    "\n",
    "from visual import Vis\n",
    "from visual import snippets_markup\n",
    "from visual import view_pycontext_output\n",
    "\n",
    "from textblob import TextBlob\n",
    "\n",
    "print('Imported pneumonia nlp utilities...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First thing, let's load our training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading annotations from file : data/training_v2.zip\n",
      "Opening local file : data/training_v2.zip\n",
      "Total Annotated Documents : 70\n"
     ]
    }
   ],
   "source": [
    "annotated_doc_map = read_doc_annotations('data/training_v2.zip')\n",
    "\n",
    "# let's also use a simple list of documents as well as this map\n",
    "annotated_docs = list(annotated_doc_map.values())\n",
    "\n",
    "print('Total Annotated Documents : {0}'.format(len(annotated_docs)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let us set up an example document to work with\n",
    "example_document = \"\"\"\n",
    "PORTABLE CHEST:  Comparison made to prior film from X:XX a.m. the same day.\n",
    "     \n",
    "The ET tube and nasogastric tube remain in good position. Cardiac and\n",
    "mediastinal contours are stable. No acute changes are seen within the lung\n",
    "parenchyma; specifically, there is no evidence of new infiltrate (skin folds\n",
    "do project over the right lung). No consolidation on either side.\n",
    "\n",
    "IMPRESSION: No evidence of pneumonia.\"\"\"\n",
    "\n",
    "example_sentence = \"\"\"IMPRESSION: No evidence of pneumonia.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our first goal is to convert the \"keywords\" that we worked on in our previous activity into pyConText \"targets\" where each target can have:\n",
    "* Standard form\n",
    "* Category \n",
    "* Additional lexical variants (with regular expressions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before we continue, note that any itemData in pyConText has 4 parts:\n",
    "1. The literal (e.g. \"pneumonia\", \"pneumoniathorax\", \"can rule out\", \"cannot be excluded\", etc)\n",
    "2. The category (e.g. \"EVIDENCE_OF_PNEUMONIA\")\n",
    "3. The regular expression (optional) used to capture the literal in the text. If no regular expression is provided, a regular expression is generated literally from the literal.\n",
    "4. The rule (optional). If the itemData is being used as a modifier, the rule states what direction the modifier operates in the sentence: current valid values are: \"forward\", the item can modify objects following it in the sentence; \"backward\", the item can modify objects preceding it in the sentence; or \"bidirectional\", the item can modify objects preceding and following it in the sentence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additionally, itemData can be instantiated in code or from a file.  Let's start with code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NodeDataView({<id> 99914448661365161132906157324590123994 </id> <phrase> pneumonia </phrase> <category> ['evidence_of_pneumonia'] </category> : {'category': 'target'}})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Now let's set up some rules for pyConText for EVIDENCE_OF_PNEUMONIA\n",
    "# At this moment, we will just set up these \"concepts\" and well handle modifiers for them after that\n",
    "\n",
    "targets1 = []\n",
    "modifiers1 = []\n",
    "\n",
    "# so before we add targets, remember from above that they will look like this : \n",
    "# targets = itemData([\"literal\", \"CATEGORY\", \"regular expression(s)\", \"empty or forward or backward or bidirectional\"])\n",
    "\n",
    "# so now let's set this up for \"pneumonia\" with the category \"EVIDENCE_OF_PNEUMONIA\"\n",
    "targets1 = itemData.itemData([\"pneumonia\", \"EVIDENCE_OF_PNEUMONIA\", \"\", \"\"])\n",
    "\n",
    "# let's go ahead and use this now on one single example sentence:\n",
    "markup = markup_sentence(example_sentence, modifiers1, targets1)\n",
    "# prettier display with IPython display\n",
    "display(markup.nodes(data = True))\n",
    "#print(markup.getXML())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function now works on entire documents combining all sentence-level objects into one object we can can then graph\n",
    "def markup_context_document(report_text, modifiers, targets):\n",
    "    context = pyConText.ConTextDocument()\n",
    "    \n",
    "    # we will use TextBlob for breaking up sentences\n",
    "    sentences = [s.raw for s in TextBlob(report_text).sentences]\n",
    "    for sentence in sentences:\n",
    "        m = markup_sentence(sentence, modifiers=modifiers, targets=targets)\n",
    "        context.addMarkup(m)\n",
    "    \n",
    "    return context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quesiton : Will we find a target match on this sentence? Will we match \"pneumonias\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_sentence_2 = \"\"\"Findings consistent with CHF, although underlying bilateral lower lobe pneumonias cannot be excluded.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NodeDataView({})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's see how things look on this sentence\n",
    "markup_sentence_2 = markup_sentence(example_sentence_2, modifiers1, targets1, verbose = True)\n",
    "display(markup_sentence_2.nodes(data = True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We didn't mark up a target for \"pneumonias\" since we only had the singular variant \"pneumonia\"\n",
    "## We can augment our targets by modifying a yaml file (.yml).  A starter yaml file is included in our course resources:\n",
    "<a href=\"../../../edit/work/decart_rule_based_nlp/KB/pneumonia_targets.yml\">KB/pneumonia_targets.yml</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Comments</th>\n",
       "      <th>Direction</th>\n",
       "      <th>Lex</th>\n",
       "      <th>Regex</th>\n",
       "      <th>Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>effusion</td>\n",
       "      <td></td>\n",
       "      <td>EVIDENCE_OF_PNEUMONIA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>pneumonia</td>\n",
       "      <td>\\bpneumonia[s]?\\b</td>\n",
       "      <td>EVIDENCE_OF_PNEUMONIA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>infiltrate</td>\n",
       "      <td></td>\n",
       "      <td>EVIDENCE_OF_PNEUMONIA</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Comments Direction         Lex              Regex                   Type\n",
       "0                       effusion                     EVIDENCE_OF_PNEUMONIA\n",
       "1                      pneumonia  \\bpneumonia[s]?\\b  EVIDENCE_OF_PNEUMONIA\n",
       "2                     infiltrate                     EVIDENCE_OF_PNEUMONIA"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import yaml\n",
    "items=[]\n",
    "pneumonia_targets_file = 'KB/pneumonia_targets.yml'\n",
    "full_targets_path =  os.path.join(os.getcwd(), pneumonia_targets_file)\n",
    "with open(full_targets_path, 'r') as f:\n",
    "    for i in (yaml.safe_load_all(f)):\n",
    "        items.append(i)\n",
    "pd.DataFrame(items)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading pneumonia;targets from : \n",
      "\t/home/jianlins/work/decart_rule_based_nlp/KB/pneumonia_targets.yml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[literal<<effusion>>; category<<['evidence_of_pneumonia']>>; re<<>>; rule<<>>,\n",
       " literal<<pneumonia>>; category<<['evidence_of_pneumonia']>>; re<<\\bpneumonia[s]?\\b>>; rule<<>>,\n",
       " literal<<infiltrate>>; category<<['evidence_of_pneumonia']>>; re<<>>; rule<<>>]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pneumonia_targets_file = 'KB/pneumonia_targets.yml'\n",
    "full_targets_path =  os.path.join(os.getcwd(), pneumonia_targets_file)\n",
    "full_targets_path =  os.path.join(os.getcwd(), pneumonia_targets_file)\n",
    "print('Loading pneumonia;targets from : \\n\\t' + full_targets_path)\n",
    "display(itemData.get_item_data(full_targets_path))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading pneumonia;targets from : /home/jianlins/work/decart_rule_based_nlp/KB/pneumonia_targets.yml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "NodeDataView({<id> 100246180146938886484983829586404969434 </id> <phrase> pneumonia </phrase> <category> ['evidence_of_pneumonia'] </category> : {'category': 'target'}, <id> 100245844219529826004192432960055544794 </id> <phrase> infiltrate </phrase> <category> ['evidence_of_pneumonia'] </category> : {'category': 'target'}})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Our first attempt was very simple target, so now let's add some additional concepts\n",
    "targets2 = []\n",
    "modifiers2 = []\n",
    "\n",
    "pneumonia_targets_file = 'KB/pneumonia_targets.yml'\n",
    "\n",
    "# so now let's set this up with more variants of \"EVIDENCE_OF_PNEUMONIA\"\n",
    "full_targets_path =  os.path.join(os.getcwd(), pneumonia_targets_file)\n",
    "print('Loading pneumonia;targets from : ' + full_targets_path)\n",
    "targets2 = itemData.get_item_data(full_targets_path)\n",
    "\n",
    "# let's go ahead and use this again on our updated targets\n",
    "context = markup_context_document(example_document, modifiers2, targets2)\n",
    "# prettier display with IPython display\n",
    "display(context.getDocumentGraph().nodes(data = True))\n",
    "#print(context.getXML())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's look at this markup in HTML with colors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p>  PORTABLE CHEST: Comparison made to prior film from X:XX a.m. the same day. The ET tube and nasogastric tube remain in good position. Cardiac and mediastinal contours are stable. No acute changes are seen within the lung parenchyma; specifically, there is no evidence of new <span style=\"font-weight: bold;color: blue;\">infiltrate</span> (skin folds do project over the right lung). No consolidation on either side. IMPRESSION: No evidence of <span style=\"font-weight: bold;color: blue;\">pneumonia</span>. </p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evidence_only_colors = {\n",
    "    \"evidence_of_pneumonia\": \"blue\"\n",
    "}\n",
    "\n",
    "context_html = mark_document_with_html(context, colors = evidence_only_colors, default_color=\"black\")\n",
    "display(HTML(context_html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"190\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "view_pycontext_output(context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's also look again to see if our regular expression for \"pneumonia\" and \"pneumonias\" worked properly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"105\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "markup_sentence_2_check = markup_context_document(example_sentence_2, modifiers2, targets2)\n",
    "view_pycontext_output(markup_sentence_2_check)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz : Targets\n",
    "In the sample code below, which one is the category?  \n",
    "\n",
    "itemData(['doctor', 'PROFESSION', '(doctor|doctors)', ''])\n",
    "\n",
    "Update the function below to pass in one of the strings below as an argument:\n",
    "* 'doctor'\n",
    "* 'PROFESSION'\n",
    "* 'doctors'\n",
    "* '' (empty string)\n",
    "* '(doctor|doctors)'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'INCORRECT.  Please try again.  See the documentation above for pyConText itemData ordering'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "identify_target_category('UPDATE_ME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz : pyConText files\n",
    "Which of the following character strings below is the delimiter (separator) for pyConText columns when reading from files by default?\n",
    "\n",
    "* ',' (comma)\n",
    "* ' ' (space)\n",
    "* '\\t' (tab)\n",
    "* '\\n' (newline)\n",
    "\n",
    "Update the function below to pass one of these strings as its argument"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'INCORRECT.  Try again.  Open the file and look at it again.  What character does it look like?'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_delimiter_quiz('UPDATE_ME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  We've  added some pyConText targets, so let's utilize them in a classifier so that we can see that adding targets can increase our Recall even if Precision may still be low\n",
    "* NOTE 1 : We will address Precision in a moment when we start working with ConText Modifiers\n",
    "* NOTE 2 : You don't need to understand all of the code below.  Essentially, it predicts pneumonia and returns 1 anytime it sees at least one target \"EVIDENCE_OF_PNEUMONIA\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************\n",
      "Performance for Classifier 1 : One total Target\n",
      "Precision : 0.6896551724137931\n",
      "Recall :    0.5882352941176471\n",
      "F1:         0.6349206349206349\n",
      "\n",
      "Confusion Matrix : \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted</th>\n",
       "      <th>False</th>\n",
       "      <th>True</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>27</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted  False  True \n",
       "Actual                 \n",
       "0             27      9\n",
       "1             14     20"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************\n",
      "Performance for Classifier 2 : 3 total Targets\n",
      "Precision : 0.5185185185185185\n",
      "Recall :    0.8235294117647058\n",
      "F1:         0.6363636363636364\n",
      "\n",
      "Confusion Matrix : \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted</th>\n",
       "      <th>False</th>\n",
       "      <th>True</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted  False  True \n",
       "Actual                 \n",
       "0             10     26\n",
       "1              6     28"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class ConTextTargetOnlyClassifier(object):\n",
    "    def __init__(self, modifiers, targets):\n",
    "        self.modifiers = modifiers\n",
    "        self.targets = targets\n",
    "    def predict(self, text):\n",
    "        # let's use our other functions in this notebook to perform sentence-wise markup and\n",
    "        # we can then check to see if these contain any EVIDENCE_OF_PNEUMONIA category types\n",
    "        context = markup_context_document(text, self.modifiers, self.targets)\n",
    "        document_graph = context.getDocumentGraph()\n",
    "        \n",
    "        # let's walk through all of the nodes in the graph and see how many are evidence of pneumonia\n",
    "        pneumonia_evidence_count = 0\n",
    "        for node in document_graph.nodes():\n",
    "            category_list = node.getCategory()\n",
    "            for category in category_list:\n",
    "                if category.upper() == 'EVIDENCE_OF_PNEUMONIA':\n",
    "                    pneumonia_evidence_count += 1\n",
    "            \n",
    "        # do we have at least one category of pneumonia evidence here?\n",
    "        return (pneumonia_evidence_count) > 0\n",
    "           \n",
    "# this one has only one target\n",
    "classifier1 = ConTextTargetOnlyClassifier(modifiers1, targets1)\n",
    "# this one has 3...\n",
    "classifier2 = ConTextTargetOnlyClassifier(modifiers2, targets2)\n",
    "\n",
    "# and now we can assess their performance\n",
    "print('****************')\n",
    "print('Performance for Classifier 1 : One total Target')\n",
    "calculate_prediction_metrics(annotated_docs, classifier1.predict)\n",
    "\n",
    "print('****************')\n",
    "print('Performance for Classifier 2 : 3 total Targets')\n",
    "calculate_prediction_metrics(annotated_docs, classifier2.predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So we have improved recall, but what are we going to do about Precision?  Since both Precision and Recall are measured equally in our F1 measure, we need to address it.  The solution to this is to improve our classification pipeline with ConText Modifiers\n",
    "* Modifiers are used to modify other tags or items in the sentence.  For example, we've already added targets for evidence of pneumonia.  We can now use modifiers to treat each of these mentions as affirmed or denied (negated) for a few examples.\n",
    "* Concretely if we had a sentence \"No evidence of pneumonia\" our current pipelines would count this as a positive case of pneumonia when it is in fact the contrary.\n",
    "* We can add a rule \"no evidence of\" to modify all targets which follow it\n",
    "* Modifiers can also modify targets which came before it in a sentence such as \"pneumonia will be ruled out\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifiers can have the following values:\n",
    "1. backward (modify any markup preceding it)\n",
    "2. forward (modify any markup following it)\n",
    "3. bidirectional (modify any markup following or preceding it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's look at a simple example of a modifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"140\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "likely_sentence = \"\"\"likely represents pneumonia\"\"\"\n",
    "\n",
    "simple_modifiers = itemData.itemData([\"likely represents\", \"PROBABLE_EXISTENCE\", \"\", \"forward\"])\n",
    "\n",
    "context_output = markup_context_document(likely_sentence, simple_modifiers, targets2)\n",
    "view_pycontext_output(context_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now let's experiment with the directionality of the modifiers (forward, backward, etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "probable_sentence_1 = \"\"\"probable case of pneumonia\"\"\"\n",
    "probable_sentence_2 = \"\"\"no evidence of pneumonia and probable arthritis\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "   \n",
    "modifiers_forward = itemData.itemData([\"probable\", \"PROBABLE_EXISTENCE\", \"\", \"forward\"])\n",
    "modifiers_backward = itemData.itemData([\"probable\", \"PROBABLE_EXISTENCE\", \"\", \"backward\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forward in this sentence modifies what we would expect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"140\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's see how we graph the relationships between targets and modifiers based on 'backward' or 'forward':\n",
    "view_pycontext_output(markup_context_document(probable_sentence_1, modifiers_forward, targets1))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In this example, nothing precedes \"probable\" so nothing to modify"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"105\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's see how we graph the relationships between targets and modifiers based on 'backward' or 'forward':\n",
    "view_pycontext_output(markup_context_document(probable_sentence_1, modifiers_backward, targets1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In this example, the forward rule does not pick up pneumonia since it comes before and we've not set up a target for arthritis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"105\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's see how we graph the relationships between targets and modifiers based on 'backward' or 'forward':\n",
    "view_pycontext_output(markup_context_document(probable_sentence_2, modifiers_forward, targets1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## And finally in this example, when we allow the target to work backward, it picks up a context where there seems to be \"no evidence of pneumonia\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"140\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's see how we graph the relationships between targets and modifiers based on 'backward' or 'forward':\n",
    "view_pycontext_output(markup_context_document(probable_sentence_2, modifiers_backward, targets1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Another important attribute that Modifiers can employ : \"terminate\"\n",
    "This allows any modifier working forward or backward to stop its modifications if it encounters one of these terms.  Let's demonstrate an example where we want \"probable\" to modify \"arthritis\" as a condition but not \"pneumonia\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_example_sentence = \"\"\"probable arthritis but not pneumonia\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp_targets = itemData.itemData([\"pneumonia\", \"EVIDENCE_OF_PNEUMONIA\", \"\", \"\"],\n",
    "                       [\"arthritis\", \"ANOTHER_CONDITION\", \"\", \"forward\"])\n",
    "\n",
    "modifiers_without_terminate = itemData.itemData([\"probable\", \"PROBABLE_EXISTENCE\", \"\", \"forward\"])\n",
    "\n",
    "modifiers_with_terminate = itemData.itemData([\"probable\", \"PROBABLE_EXISTENCE\", \"\", \"forward\"],\n",
    "                                   [\"but\", \"CONJ\", \"\", \"terminate\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Without the \"terminate\" modifier, \"probable_existence\" is applied to both arthritis and also pneumonia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"175\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "view_pycontext_output(markup_context_document(terminate_example_sentence, modifiers_without_terminate, temp_targets))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Here with the \"terminate\" modifier \"probable_existence\" is applied only to \"arthritis\" and does not modify beyond the conjunction \"but\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"210\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "view_pycontext_output(markup_context_document(terminate_example_sentence, modifiers_with_terminate, temp_targets))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifier Quiz : Directionality\n",
    "Imagine that you are given the string below as a modifier.  What directionality do you think it should have as you are setting up the modifier?\n",
    "\n",
    "'was ruled out'\n",
    "\n",
    "Update the function below to pass in one of the following values:\n",
    "* 'forward'\n",
    "* 'backward'\n",
    "* 'bidirectional'\n",
    "* 'terminate'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'INCORRECT.  It is not clear what you passed in.  Please see the list of possible answers'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "modifier_directionality_quiz('UPDATE_ME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now let's get back to how these are targets and modifiers are used together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Modifiers Loaded for pipeline #3 : [8]\n",
      "Total Targets Loaded for pipeline #3 : [3]\n"
     ]
    }
   ],
   "source": [
    "pneumonia_modifiers_file = 'KB/pneumonia_modifiers.yml'\n",
    "modifier_file_path =  os.path.join(os.getcwd(), pneumonia_modifiers_file)\n",
    "modifiers3 = itemData.get_item_data(modifier_file_path)\n",
    "# let's just use the same targets as above for our third pipeline\n",
    "targets3 = targets2\n",
    "\n",
    "\n",
    "print('Total Modifiers Loaded for pipeline #3 : [{0}]'.format(len(modifiers3)))\n",
    "print('Total Targets Loaded for pipeline #3 : [{0}]'.format(len(targets3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now we can use leverage both Targets and Modifiers to properly leverage context let's see what this looks like in HTML with our document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare some colors for displaying any markup we might see\n",
    "colors = {\n",
    "    \"evidence_of_pneumonia\": \"blue\",\n",
    "    \"definite_negated_existence\": \"red\",\n",
    "    \"probable_negated_existence\": \"indianred\",\n",
    "    \"ambivalent_existence\": \"blue\",\n",
    "    \"probable_existence\": \"forestgreen\",\n",
    "    \"definite_existence\": \"green\",\n",
    "    \"historical\": \"goldenrod\",\n",
    "    \"indication\": \"pink\",\n",
    "    \"acute\": \"golden\"\n",
    "}\n",
    "\n",
    "# let's mark up a new context object for our pipeline#3\n",
    "context3 = markup_context_document(example_document, modifiers3, targets3)\n",
    "\n",
    "display(HTML(mark_document_with_html(context3, colors = colors, default_color=\"black\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And use Brat style display:\n",
    "view_pycontext_output(context3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OK, so now that we've got a few Targets and Modifiers to start from, let's process all of the documents and then visualize the relationships between Targets and Modifiers for some of these documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# NOTE : This is a \"magic\" command to Jupyter to time the execution of this entire cell\n",
    "\n",
    "report_results = []\n",
    "print('Marking up all documents...')\n",
    "for anno_doc in annotated_docs:\n",
    "    report_context = markup_context_document(anno_doc.text, modifiers3, targets3)\n",
    "    # package this up into a class that the RadNLP utilities can use\n",
    "    results = classrslts(context_document=report_context, exam_type=\"Chest X-Ray\", report_text=anno_doc.text, classification_result='N/A')\n",
    "    report_results.append(results)\n",
    "    \n",
    "print('DONE Marking up all documents...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function let's us iterate through all documents and view the markup\n",
    "def view_pycontext_graph(class_results):\n",
    "    @interact(i=ipywidgets.IntSlider(min=0, max=len(class_results)-1))\n",
    "    def _view_markup(i):\n",
    "        class_result = class_results[i]\n",
    "        view_pycontext_output(class_result.context_document)\n",
    "        \n",
    "        \n",
    "# This function let's us iterate through all documents and view the markup\n",
    "def view_annotation_markup(anno_docs, colors):\n",
    "    @interact(i=ipywidgets.IntSlider(min=0, max=len(anno_docs)-1))\n",
    "    def _view_markup(i):\n",
    "        report_html = pneumonia_annotation_html_markup(anno_docs[i])\n",
    "        \n",
    "        display(HTML(report_html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "view_pycontext_graph(report_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Now that we can see how modifiers can interact with Targets, this will be a useful tool to help us increase Precision by ignoring False Positives like \"no evidence of pneumonia\".\n",
    "\n",
    "## Next, Jianlin will talk about a way to set up classification rules combining Targets and Modifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quiz: Frequent Modifier categories\n",
    "Earlier we grouped and counted modifiers by their category.  Besides the modifiers for conjuction ('CONJ') which is the most frequent modifier category, what is the second most common modifier category in the set provided to you?\n",
    "\n",
    "* 'PROBABLE_NEGATED_EXISTENCE'\n",
    "* 'PROBABLE_EXISTENCE'\n",
    "* 'DEFINITE_NEGATED_EXISTENCE'\n",
    "* 'HISTORICAL'\n",
    "\n",
    "Update the function below to pass in the most common modifier category.  Hint : You may need to scroll above to see this output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "second_most_frequent_modifier_quiz('UPDATE_ME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/><br/>This material presented as part of the DeCART Data Science for the Health Science Summer Program at the University of Utah in 2019.<br/>\n",
    "Presenters : Dr. Wendy Chapman, Kelly Peterson, Alec Chapman, Jianlin Shi <br> Acknowledgement: Many thanks to Olga Patterson because part of the materials are adopted from his previous work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}