intro/Intro_Supervision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"imgs/fonduer-logo.png\" width=\"100px\" style=\"margin-right:20px\">\n",
    "\n",
    "# Tutorial: Providing Supervision using Labeling Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running locally?\n",
    "\n",
    "If you're running this tutorial interactively on your own machine, you'll need to create a new PostgreSQL database named `intro_supervision`.\n",
    "\n",
    "If you already have the database `intro_supervision` in your postgresql, please uncomment the first line to drop it. Otherwise, download our database snapshots by executing `./download_data.sh` in the intro tutorial directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#! dropdb --if-exists intro_supervision\n",
    "! createdb intro_supervision\n",
    "! psql intro_supervision < data/intro_supervision.sql > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Providing Supervision by Writing Labeling Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, you will learn what a labeling function (LF) is and how to write them by leverage Fonduer's [data model utilities](https://fonduer.readthedocs.io/en/stable/user/data_model_utils.html).\n",
    "\n",
    "At a high level, a labeling function is a simple Python function that takes a candidate (a part and numerical value, in these intro tutorials) as input, and returns a label for the input candidate. Labels can be one of these values: {-1, 0, 1}. A label of -1 signifies that a candidate is False, 0 is a way to abstain from voting, and +1 labels the candidate as True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2019-04-01 22:32:39,498][INFO] fonduer.meta:50 - Setting logging directory to: logs/2019-04-01_22-32-39\n",
      "[2019-04-01 22:32:39,544][INFO] fonduer.meta:134 - Connecting user:None to localhost:5432/intro_supervision\n",
      "[2019-04-01 22:32:40,722][INFO] fonduer.meta:161 - Initializing the storage schema\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "import sys\n",
    "import logging\n",
    "\n",
    "ATTRIBUTE = \"intro_supervision\"\n",
    "conn_string = f'postgresql://localhost:5432/{ATTRIBUTE}'\n",
    "\n",
    "from fonduer import Meta, init_logging\n",
    "\n",
    "# Configure logging for Fonduer\n",
    "init_logging(log_dir=\"logs\")\n",
    "\n",
    "session = Meta.init(conn_string).Session()\n",
    "\n",
    "from fonduer.candidates.models import candidate_subclass, mention_subclass\n",
    "\n",
    "Part = mention_subclass(\"Part\")\n",
    "Attr = mention_subclass(\"Attr\")\n",
    "PartAttr = candidate_subclass(\"PartAttr\", [Part, Attr])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I. Background\n",
    "\n",
    "### Using a Development Set to Evaluate our Supervision\n",
    "For convenience in error analysis and evaluation, we have already annotated the dev and test set for this tutorial, and we'll now load it using an externally-defined helper function. If you're interested in the example implementation details, please see the script we now load:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading 2533 candidate labels\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "11e8d174121149819e1f63b191f9013a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=2533), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "GoldLabels created: 2533\n"
     ]
    }
   ],
   "source": [
    "from hardware_utils import load_hardware_labels\n",
    "\n",
    "gold_file = 'data/hardware_tutorial_gold.csv'\n",
    "load_hardware_labels(session, PartAttr, gold_file, ATTRIBUTE ,annotator_name='gold')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading Candidates\n",
    "\n",
    "Next, we can get our train and development set candidates by issuing SQLAlchemy queries for the `Part_Attr` candidate we defined during candidate generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training candidates: 2533\n"
     ]
    }
   ],
   "source": [
    "train_cands = sorted(session.query(PartAttr).all())\n",
    "\n",
    "print(f\"Number of training candidates: {len(train_cands)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing Labeling Functions \n",
    "\n",
    "Supervisions can be in different sources such as patterns or heuristics. Fonduer uses labeling functions to encode these supervisions that can be used to distinguish whether or not a candidate is true or false. In this notebook, we will describe how to use Fonduer API to express supervision via different modal signals.\n",
    "\n",
    "The full list of functions that you can use are documented here:\n",
    "\n",
    "https://fonduer.readthedocs.io/en/stable/user/data_model_utils.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.utils.data_model_utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recall: what's in a candidate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "cand = train_cands[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at part number first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "part object:                      Part(SpanMention(\"2N3904\", sentence=5396, chars=[24,29], words=[3,3]))\n",
      "part text:                        2N3904\n",
      "part sentence object:             Sentence (Doc: 'AUKCS04635-1', Sec: 0, Par: 10, Idx: 10, Text: 'Complementary pair with 2N3904')\n",
      "part sentence text:               Complementary pair with 2N3904\n",
      "check if part is in a table:      False\n",
      "check if part has in visual info: True\n"
     ]
    }
   ],
   "source": [
    "print(f\"part object:                      {cand.part}\")\n",
    "print(f\"part text:                        {cand.part.context.get_span()}\")\n",
    "print(f\"part sentence object:             {cand.part.context.sentence}\")\n",
    "print(f\"part sentence text:               {cand.part.context.sentence.text}\")\n",
    "print(f\"check if part is in a table:      {cand.part.context.sentence.is_tabular()}\")\n",
    "print(f\"check if part has in visual info: {cand.part.context.sentence.is_visual()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can look at the `attr`, which is the number representing the maximum collector-emitter voltage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "attr object:                      Attr(SpanMention(\"150\", sentence=13054, chars=[0,2], words=[0,0]))\n",
      "attr text:                        150\n",
      "attr sentence object:             Sentence (Doc: 'AUKCS04635-1', Table: 0, Row: 6, Col: 2, Index: 58, Text: '150')\n",
      "attr sentence text:               150\n",
      "check if attr is in a table:      True\n",
      "check if attr has in visual info: True\n"
     ]
    }
   ],
   "source": [
    "print(f\"attr object:                      {cand.attr}\")\n",
    "print(f\"attr text:                        {cand.attr.context.get_span()}\")\n",
    "print(f\"attr sentence object:             {cand.attr.context.sentence}\")\n",
    "print(f\"attr sentence text:               {cand.attr.context.sentence.text}\")\n",
    "print(f\"check if attr is in a table:      {cand.attr.context.sentence.is_tabular()}\")\n",
    "print(f\"check if attr has in visual info: {cand.attr.context.sentence.is_visual()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: Write a labeling function to check if two mentions in one candidate are in the same page. \n",
    "If they are, label the candidate True, otherwise, label it False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "ABSTAIN = 0\n",
    "FALSE = 1\n",
    "TRUE = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def LF_same_page(c):\n",
    "    return TRUE if same_page(c) else FALSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Try again.\n"
     ]
    }
   ],
   "source": [
    "# Sanity check: the previous labeling function should pass the follwoing test.\n",
    "true_candidate = train_cands[81]\n",
    "false_candidate = train_cands[10]\n",
    "\n",
    "if (LF_same_page(true_candidate) == TRUE and LF_same_page(false_candidate) == FALSE):\n",
    "    print(\"You passed!\")\n",
    "else:\n",
    "    print(\"Try again.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: Write a labeling function based on your insight of the data.\n",
    "\n",
    "For example, inspecting several documents may reveal that storage temperatures are typically listed inside a table where the row header contains the word \"storage\". This intuitive pattern can be directly expressed as a labeling function. Similarly, the word \"temperature\" is an obvious positive signal.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def LF_storage_row(c):\n",
    "    return TRUE if 'storage' in get_row_ngrams(c.attr) else ABSTAIN\n",
    "\n",
    "def LF_temperature_row(c):\n",
    "    return TRUE if 'temperature' in get_row_ngrams(c.attr) else ABSTAIN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3: Write a labeling function based on alignment information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def LF_collector_aligned(c):\n",
    "    return FALSE if overlap(\n",
    "        ['collector', 'collector-current', 'collector-base', 'collector-emitter'],\n",
    "        list(get_aligned_ngrams(c.attr))) else ABSTAIN\n",
    "\n",
    "def LF_current_aligned(c):\n",
    "    ngrams = get_aligned_ngrams(c.attr)\n",
    "    return FALSE if overlap(\n",
    "        ['current', 'dc', 'ic'],\n",
    "        list(get_aligned_ngrams(c.attr))) else ABSTAIN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then collect all of these labeling functions in a list which we will provide to Fonduer as supervision signals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "LFs = [\n",
    "    LF_same_page,\n",
    "    LF_storage_row,\n",
    "    LF_temperature_row,\n",
    "    LF_collector_aligned,\n",
    "    LF_current_aligned\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying the Labeling Functions\n",
    "\n",
    "Next, we need to actually run the LFs over all of our training candidates, producing a set of `Labels` and `LabelKeys` (just the names of the LFs) in the database. We'll do this using the `Labeler`. Note that this will delete any existing `Labels` and `LabelKeys` for this candidate set.\n",
    "\n",
    "View the API provided by the `Labeler` on [ReadTheDocs](https://fonduer.readthedocs.io/en/stable/user/supervision.html#fonduer.supervision.Labeler)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2019-04-01 22:32:49,661][INFO] fonduer.supervision.labeler:219 - Clearing Labels (split 0)\n",
      "[2019-04-01 22:32:49,669][INFO] fonduer.utils.udf:54 - Running UDF...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c77dea62867402485b7eb8d79d75c8a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "CPU times: user 5.87 s, sys: 168 ms, total: 6.04 s\n",
      "Wall time: 7.25 s\n",
      "CPU times: user 3.47 s, sys: 132 ms, total: 3.6 s\n",
      "Wall time: 4.6 s\n"
     ]
    }
   ],
   "source": [
    "from fonduer.supervision import Labeler\n",
    "\n",
    "labeler = Labeler(session, [PartAttr])\n",
    "\n",
    "%time labeler.apply(split=0, lfs=[LFs], train=True)\n",
    "%time L_train = labeler.get_label_matrices([train_cands])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Labeling Function Metrics\n",
    "\n",
    "Next, we can view insights provided by Fonduer to better understand the quality and coverage of our labeling functions.\n",
    "\n",
    "In order to view statistics about the resulting label matrix, we provide several metrics to evaluate labelding functions:\n",
    "* **Coverage** is the fraction of candidates that the labeling function emits a non-zero label for.\n",
    "* **Overlap** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a non-zero label for.\n",
    "* **Conflict** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a conflicting non-zero label for.\n",
    "* **TP** is the number of True Positive candidates, or true candidates which were correctly labeled as True.\n",
    "* **FP** is the number of False Positive candidates, or false candidates which were incorrectly labeled as True.\n",
    "* **FN** is the number of False Negative candidates, or true candidates which were incorrectly labeled as False.\n",
    "* **TN** is the number of True Negative candidates, or false candidates which were correctly labeled as False.\n",
    "\n",
    "In addition, because we have already loaded the gold labels, we can view the emperical accuracy of these labeling functions when compared to our gold labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "L_gold_dev = labeler.get_gold_labels([train_cands], annotator='gold')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>j</th>\n",
       "      <th>Polarity</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Overlaps</th>\n",
       "      <th>Conflicts</th>\n",
       "      <th>Correct</th>\n",
       "      <th>Incorrect</th>\n",
       "      <th>Emp. Acc.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LabelKey (LF_collector_aligned)</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.015002</td>\n",
       "      <td>0.015002</td>\n",
       "      <td>0.003553</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LabelKey (LF_current_aligned)</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.168180</td>\n",
       "      <td>0.168180</td>\n",
       "      <td>0.075010</td>\n",
       "      <td>426</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LabelKey (LF_same_page)</th>\n",
       "      <td>2</td>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.906830</td>\n",
       "      <td>0.240032</td>\n",
       "      <td>0.135413</td>\n",
       "      <td>1958</td>\n",
       "      <td>339</td>\n",
       "      <td>0.852416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LabelKey (LF_storage_row)</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0.070272</td>\n",
       "      <td>0.070272</td>\n",
       "      <td>0.058824</td>\n",
       "      <td>0</td>\n",
       "      <td>178</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LabelKey (LF_temperature_row)</th>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0.071852</td>\n",
       "      <td>0.071852</td>\n",
       "      <td>0.060403</td>\n",
       "      <td>0</td>\n",
       "      <td>182</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 j Polarity  Coverage  Overlaps  Conflicts  \\\n",
       "LabelKey (LF_collector_aligned)  0        1  0.015002  0.015002   0.003553   \n",
       "LabelKey (LF_current_aligned)    1        1  0.168180  0.168180   0.075010   \n",
       "LabelKey (LF_same_page)          2   [1, 2]  0.906830  0.240032   0.135413   \n",
       "LabelKey (LF_storage_row)        3        2  0.070272  0.070272   0.058824   \n",
       "LabelKey (LF_temperature_row)    4        2  0.071852  0.071852   0.060403   \n",
       "\n",
       "                                 Correct  Incorrect  Emp. Acc.  \n",
       "LabelKey (LF_collector_aligned)       38          0   1.000000  \n",
       "LabelKey (LF_current_aligned)        426          0   1.000000  \n",
       "LabelKey (LF_same_page)             1958        339   0.852416  \n",
       "LabelKey (LF_storage_row)              0        178   0.000000  \n",
       "LabelKey (LF_temperature_row)          0        182   0.000000  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from metal import analysis\n",
    "\n",
    "analysis.lf_summary(L_train[0], lf_names=labeler.get_keys(), Y=L_gold_dev[0].todense().reshape(-1,).tolist()[0])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}