10_NLP_DocumentClassification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# From Mention-Level Annotations to Document Classification\n",
    "\n",
    "## 1. Why do we need document classification?\n",
    "\n",
    "Think about a case with multiple mentions in one document. How do we decide the document level conclusion when these mentions have \"conflicted\" information? For example, \n",
    "\n",
    ">Small **left pleural effusion**. **Right pleural effusion can be excluded**.\n",
    "\n",
    "In this example, should we conclude that the report indicates pneumonia or does not indicate pneumonia?\n",
    "\n",
    "There are many other situations that we need to draw a document level conclusion based on multiple mention level annotations. Certainly, we can train a machine learning classifier to accomplish this task, which you will learn in another class. But here we are going to learn how to do it in rule-based way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Restore from where we are using pyConText"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import everything that we will need\n",
    "import pyConTextNLP\n",
    "from pyConTextNLP import pyConText\n",
    "from pyConTextNLP.itemData import itemData\n",
    "from pyConTextNLP.utils import get_document_markups\n",
    "from pyConTextNLP.display.html import mark_document_with_html\n",
    "import os\n",
    "import os.path\n",
    "from nlp_pneumonia_utils import Annotation\n",
    "from nlp_pneumonia_utils import AnnotatedDocument\n",
    "from nlp_pneumonia_utils import read_brat_annotations\n",
    "from nlp_pneumonia_utils import read_annotations\n",
    "from nlp_pneumonia_utils import calculate_prediction_metrics\n",
    "from nlp_pneumonia_utils import markup_context_document\n",
    "from DocumentClassifier import DocumentClassifier\n",
    "from IPython.display import display, HTML, Image\n",
    "\n",
    "from visual import Vis\n",
    "from visual import snippets_markup\n",
    "from visual import view_pycontext_output\n",
    "from visual import convertMarkups2DF\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's just consider the example at the beginning as a document,\n",
    "# and run pyConText to get markups\n",
    "\n",
    "report = \"Right pleural effusion can be excluded. Likely small left pleural effusion. \"\n",
    "\n",
    "targets = itemData([\"effusion\", \"EVIDENCE_OF_PNEUMONIA\", r\"effusion[s]?\", \"\"])\n",
    "\n",
    "modifiers = pyConTextNLP.itemData.instantiateFromCSVtoitemData(os.path.join(os.getcwd(),'KB/pneumonia_modifiers.tsv'))\n",
    "\n",
    "markups=markup_context_document(report,modifiers,targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"235\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# To confirm what we get from pyConText\n",
    "view_pycontext_output(markups)\n",
    "        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Use DocumentClassifier to define the rules for document classification\n",
    "\n",
    "After we processed a document, we will get a list of pyConText output. But this is not the end. We want to conclude whether this document is pneumonia positive or not. That's where the DocumentClassifier comes in.\n",
    "\n",
    "### 3.1 Simple document classification rules\n",
    "First, let's start from two simple cases:\n",
    "1. If there is a true mention of pneumonia evidence, we should conclude \"PNEUMONIA_DOC_YES\"\n",
    "2. Otherwise, we should conclude \"PNEUMONIA_DOC_NO\"\n",
    "\n",
    "\n",
    "We can easily write some python code for that. Here we do something more----externalize the rule definitions, so that we can directly reuse our code for other projects. Here is an example rule file [doc_inferences.csv](../../../edit/work/decart_rule_based_nlp/KB/doc_inferences.csv):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "DocConclusion,EvidenceTypes\n",
    "# The rule in document inferences are processed from top to bottom.\n",
    "# If any one of the rules is matched, the rest rules below it will be skipped.\n",
    "# if the document has a EVIDENCE_OF_PNEUMONIA annotation, conclude PNEUMONIA_DOC_YES.\n",
    "PNEUMONIA_DOC_YES,EVIDENCE_OF_PNEUMONIA\n",
    "# if no above rule matched, conclude NEG_COLON_CA_DOC (default conclusion)\n",
    "PNEUMONIA_DOC_NO\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 How to integrate pyConText processed results\n",
    "However, the rules above haven't make any use of pyConText outputs. We definitely want to exclude the evidences that are negated. A simple idea to fix it is to go over all the annotations, change the negated \"EVIDENCE_OF_PNEUMONIA\" annotations to another annotation type, and then our DocumentClassifier can use the rules above to draw the conclusions correctly.\n",
    "\n",
    "Thus, we need another inference component (feature inferencer) to make this change. Its rule ([featurer_inferences.csv](../../../edit/work/decart_rule_based_nlp/KB/featurer_inferences.csv)) format is like following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "ConclusionType,SourceType,ModifierValues\n",
    "#if an annotation is 'EVIDENCE_OF_PNEUMONIA', only has a modifier \"DEFINITE_NEGATED_EXISTENCE', change this annotation type to NEG_EVIDENCE\n",
    "NEG_EVIDENCE,EVIDENCE_OF_PNEUMONIA,DEFINITE_NEGATED_EXISTENCE\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Dissect DocumentClassifier step by step\n",
    "To use these two sets of rules, we need to initiate our DocumentClassifier. This DocumentClassifier also wrapped pyConText inside to make it easier to use:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_doc_type='PNEUMONIA_DOC_YES'\n",
    "TARGETS_FILE_PATH = 'KB/pneumonia_targets.tsv'\n",
    "MODIFIERS_FILE_PATH = 'KB/pneumonia_modifiers.tsv'\n",
    "FEATURE_INFERENCER_FILE_PATH = 'KB/featurer_inferences.csv'\n",
    "DOC_INFERENCER_FILE_PATH = 'KB/doc_inferences.csv'\n",
    "# clear just in case files/regular expressions have been updated\n",
    "classifier = DocumentClassifier(TARGETS_FILE_PATH, MODIFIERS_FILE_PATH,\n",
    "                               FEATURE_INFERENCER_FILE_PATH, DOC_INFERENCER_FILE_PATH,\n",
    "                               {pos_doc_type})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's check how this \"classifier\" gets to the conclusion step by step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\t\t\t      <iframe src = \"tmp/x.html\" frameborder=\"0\" width = \"850\" height = \"235\">\n",
       "\t\t\t         Sorry your browser does not support inline frames.\n",
       "\t\t\t      </iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 1st use pyConText to process the input text, \n",
    "context_doc = markup_context_document(report, classifier.modifiers, classifier.targets)\n",
    "view_pycontext_output(context_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read out the pyConText output into dataframe format\n",
    "annotations, relations, doc_txt = convertMarkups2DF(get_document_markups(context_doc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>markup_id</th>\n",
       "      <th>vis_category</th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "      <th>txt</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>T0</td>\n",
       "      <td>Target</td>\n",
       "      <td>14</td>\n",
       "      <td>22</td>\n",
       "      <td>effusion</td>\n",
       "      <td>evidence_of_pneumonia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>T1</td>\n",
       "      <td>Modifier</td>\n",
       "      <td>29</td>\n",
       "      <td>38</td>\n",
       "      <td>excluded</td>\n",
       "      <td>definite_negated_existence</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>T2</td>\n",
       "      <td>Target</td>\n",
       "      <td>66</td>\n",
       "      <td>74</td>\n",
       "      <td>effusion</td>\n",
       "      <td>evidence_of_pneumonia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>T3</td>\n",
       "      <td>Modifier</td>\n",
       "      <td>40</td>\n",
       "      <td>46</td>\n",
       "      <td>Likely</td>\n",
       "      <td>probable_existence</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  markup_id vis_category start end        txt                        type\n",
       "0        T0       Target    14  22   effusion       evidence_of_pneumonia\n",
       "1        T1     Modifier    29  38   excluded  definite_negated_existence\n",
       "2        T2       Target    66  74   effusion       evidence_of_pneumonia\n",
       "3        T3     Modifier    40  46     Likely          probable_existence"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's see what is in annotations:\n",
    "annotations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>relation_id</th>\n",
       "      <th>type</th>\n",
       "      <th>arg1_cate</th>\n",
       "      <th>arg1_id</th>\n",
       "      <th>arg2_cate</th>\n",
       "      <th>arg2_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>R0</td>\n",
       "      <td>definite_negated_existence</td>\n",
       "      <td>Modifier</td>\n",
       "      <td>T1</td>\n",
       "      <td>Target</td>\n",
       "      <td>T0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>R1</td>\n",
       "      <td>probable_existence</td>\n",
       "      <td>Modifier</td>\n",
       "      <td>T3</td>\n",
       "      <td>Target</td>\n",
       "      <td>T2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  relation_id                        type arg1_cate arg1_id arg2_cate arg2_id\n",
       "0          R0  definite_negated_existence  Modifier      T1    Target      T0\n",
       "1          R1          probable_existence  Modifier      T3    Target      T2"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And what is in \"relations\":\n",
    "relations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we use feature inference rules to change our negated annotations' type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pos_evidence', 'pos_evidence', 'pos_evidence', 'pos_evidence']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matched_conclusion_types = classifier.feature_inferencer.process(annotations, relations)\n",
    "matched_conclusion_types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the 1st annotation type has been changed to 'neg_evidence'. Next, we can draw document level conclusion by simply check if there still is any annotation of 'EVIDENCE_OF_PNEUMONIA':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'pneumonia_doc_no'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_conclusion = classifier.document_inferencer.process(matched_conclusion_types)\n",
    "doc_conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 Wrap up\n",
    "To make the call even simpler, DocumentClassifier has already wrapped up all the codes above into a single function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_conclusion = classifier.classify_doc(report)\n",
    "doc_conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize the last document that has been processed by it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "view_pycontext_output(classifier.get_last_context_doc())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 4. Excercise\n",
    "Let's try to switch the sentences in the example\n",
    "See what happens. Does the order of mention-level annotation affects final conclusion?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here:\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Quiz\n",
    "Let's try a few questions, see if you've understood the content of this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from quiz_utils import doc_classify_1\n",
    "doc_classify_1()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from quiz_utils import doc_classify_2\n",
    "doc_classify_2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from quiz_utils import doc_classify_3\n",
    "doc_classify_3()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<br/><br/>This material presented as part of the DeCART Data Science for the Health Science Summer Program at the University of Utah in 2019.<br/>\n",
    "Presenters : Dr. Wendy Chapman, Kelly Peterson, Alec Chapman, Jianlin Shi <br> Acknowledgement: Many thanks to Olga Patterson because part of the materials are adopted from his previous work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}