27_Calculate_Agreement_for_Brat_Annotations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculating Agreement for Brat annotations\n",
    "\n",
    "Now we have your annotations ready and have learned the agreement formulas, let's try some exercises to calculate the agreement betwee each other.\n",
    "\n",
    "Although the formulas are simple, efficiently getting the numbers in the contingency table is not trivial. We have provided an optimized function for you here (If you are interested how we implemented it, check [here](./compare_utils.py). ). Let's try it out.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting intervaltree\n",
      "  Downloading https://files.pythonhosted.org/packages/e8/f9/76237755b2020cd74549e98667210b2dd54d3fb17c6f4a62631e61d31225/intervaltree-3.0.2.tar.gz\n",
      "Requirement already satisfied: sortedcontainers<3.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from intervaltree) (2.1.0)\n",
      "Building wheels for collected packages: intervaltree\n",
      "  Building wheel for intervaltree (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/jianlins/.cache/pip/wheels/08/99/c0/5a5942f5b9567c59c14aac76f95a70bf11dccc71240b91ebf5\n",
      "Successfully built intervaltree\n",
      "Installing collected packages: intervaltree\n",
      "Successfully installed intervaltree-3.0.2\n"
     ]
    }
   ],
   "source": [
    "!pip install intervaltree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import packages\n",
    "import os\n",
    "from compare_utils import compare_projects,show_annotations\n",
    "from IPython.display import HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Initiate the directories and read the annotations\n",
    "\n",
    "First, we need to tell compare who against who. In Brat, annotations are saved in directories, thus the question is equivalent to compare which directory against which.\n",
    "\n",
    "If you are not sure what directories you should look for, check the list here:\n",
    "https://brat.jupyter.med.utah.edu/#/student_folders/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tell where is the projects located, you need to replace them with your project name and reference project name\n",
    "import getpass\n",
    "annotator_a=getpass.getuser()\n",
    "annotator_b='goldstandard'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the project name to real directory path\n",
    "\n",
    "brat_projects_loc=os.path.join(os.path.expanduser('~'),'BRAT')\n",
    "annotator_a_dir=os.path.join(brat_projects_loc, annotator_a)\n",
    "annotator_b_dir=os.path.join(brat_projects_loc, annotator_b)\n",
    "\n",
    "# you could try to print annotator_a and annotator_b out to see where they are\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Strict comparison\n",
    "\n",
    "**compare_projects** is the function that we wrapped up the meat in. It takes in 2~3 paramters:\n",
    "1. Your directory \n",
    "2. The directory that you want to compare against\n",
    "3. compare method ('strict' or 'relax')\n",
    "\n",
    "It turns a dictionary of evaluators with annotation types as the key, an Evaluator as the value. The Evaluator class will contain all the numbers in the contingency table we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_map, evaluators = compare_projects(annotator_a_dir, annotator_b_dir, 'strict')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**compare_projects** returns two values:\n",
    "1. *doc_map* contains a dictionary that maps a document name to its content text\n",
    "2. *evaluators* contains a dictionary that maps an annotation type to the corresponding compared results--an object of [Evaluator](./compare_utils.py)\n",
    "\n",
    "Next, let's take a look at what's inside evaluators:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EVIDENCE_OF_PNEUMONIA\n",
      "0 0 14 None\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B+</th>\n",
       "      <th>B-</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A+</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A-</th>\n",
       "      <td>14</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    B+   B-\n",
       "A+   0  0.0\n",
       "A-  14  NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PNEUMONIA_DOC_NO\n",
      "0 0 4 None\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B+</th>\n",
       "      <th>B-</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A+</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A-</th>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    B+   B-\n",
       "A+   0  0.0\n",
       "A-   4  NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PNEUMONIA_DOC_YES\n",
      "0 0 6 None\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B+</th>\n",
       "      <th>B-</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A+</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A-</th>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    B+   B-\n",
       "A+   0  0.0\n",
       "A-   6  NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for type_name, evaluator in evaluators.items():\n",
    "    print(type_name)\n",
    "    a,b,c,d=evaluator.get_values()\n",
    "#   now you can print these numbers\n",
    "    print(a,b,c,d)\n",
    "#   or display in a contingency table\n",
    "    display(evaluator.display_values())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can caculate your IAA:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code goes here:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Relaxed comparsion\n",
    "When comparin mention level annotations, it is more useful to use relaxed comparision -- consider a match if an annotation of annotator A overlaps with the annotator B's. For instance, \"Left lower lobe pneumonia\" vs \"pneumonia\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the code is very similar to the above\n",
    "doc_map,evaluators = compare_projects(annotator_a_dir, annotator_b_dir, 'relax')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EVIDENCE_OF_PNEUMONIA\n",
      "0 0 14 None\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B+</th>\n",
       "      <th>B-</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A+</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A-</th>\n",
       "      <td>14</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    B+   B-\n",
       "A+   0  0.0\n",
       "A-  14  NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PNEUMONIA_DOC_NO\n",
      "0 0 4 None\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B+</th>\n",
       "      <th>B-</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A+</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A-</th>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    B+   B-\n",
       "A+   0  0.0\n",
       "A-   4  NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PNEUMONIA_DOC_YES\n",
      "0 0 6 None\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B+</th>\n",
       "      <th>B-</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A+</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A-</th>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    B+   B-\n",
       "A+   0  0.0\n",
       "A-   6  NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for type_name, evaluator in evaluators.items():\n",
    "    print(type_name)\n",
    "    a,b,c,d=evaluator.get_values()\n",
    "#   now you can print these numbers\n",
    "    print(a,b,c,d)\n",
    "#   or display in a contingency table\n",
    "    display(evaluator.display_values())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can try to calculate your IAA:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you only want to compare some types, here is the code you can use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_map,evaluators = compare_projects(annotator_a_dir, annotator_b_dir, 'relax',['PNEUMONIA_DOC_NO','PNEUMONIA_DOC_YES'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code goes here:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Show the disagreement\n",
    "\n",
    "Now we are wondering where are the disagreement annotations. Evaluator saved that information as well. Let's try to display them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Show the annotations in annotator_a, but not annotator_b (false positive)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PNEUMONIA_DOC_NO\n",
      "(0, 0, 4, None)\n",
      "\tNo documents to display.\n",
      "PNEUMONIA_DOC_YES\n",
      "(0, 0, 6, None)\n",
      "\tNo documents to display.\n"
     ]
    }
   ],
   "source": [
    "for type_name, evaluator in evaluators.items():\n",
    "    print(type_name)\n",
    "    print(evaluator.get_values())\n",
    "    fps=evaluator.get_fps()\n",
    "    show_annotations(fps, doc_map,annotator_a,annotator_b,900,200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Show the annotations in annotator_b, but not annotator_a (false negative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PNEUMONIA_DOC_NO\n",
      "(0, 0, 4, None)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "82417276967f4ff59e08e62fdeef3fa8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HTML(value='<html><div style=\"background-color:#f9f9f9;padding-left:10px;width: 877px; \"><table width=100% ><c…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PNEUMONIA_DOC_YES\n",
      "(0, 0, 6, None)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d87f8fef5f174bc6a91ac3a39dabdcc1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HTML(value='<html><div style=\"background-color:#f9f9f9;padding-left:10px;width: 877px; \"><table width=100% ><c…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for type_name, evaluator in evaluators.items():\n",
    "    print(type_name)\n",
    "    fns=evaluator.get_fns()\n",
    "    print(evaluator.get_values())\n",
    "    show_annotations(fns, doc_map,annotator_a,annotator_b,900,200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/><br/>This material presented as part of the DeCART Data Science for the Health Science Summer Program at the University of Utah in 2019.<br/>\n",
    "Presenters : Dr. Wendy Chapman, Kelly Peterson, Alec Chapman, Jianlin Shi <br> Acknowledgement: Many thanks to Olga Patterson because part of the materials are adopted from his previous work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}