08_NLP_Regex_for_Concept_Extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Regular Expression for Concept Extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language\n",
    "\n",
    "With language, we use a large, but finite vocabulary to describe the world. These descriptions are composed of words and phrases that can be predicted with some regularity. For instance, when describing 'infiltration', we can use phrases like  'infiltration', 'infiltrate', 'infiltrated', etc. written using letters that are placed together with some regularity. We can readily identify the infiltration written with these three expressions by identifying the pattern for constructing these lexical variants. What do these three words have in common? \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$${\\bf infiltrat}e$$\n",
    "$${\\bf infiltrat}ed$$\n",
    "$${\\bf infiltrat}ion$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representing patterns\n",
    "\n",
    "Regular expressions aka regex are string statements that represent the regularity in the text that you would like to leverage to **match** or **extract** a concept. Regex is a powerful and commonly used tool for natural language processing and has been used for several tasks:\n",
    "\n",
    "* identify and classify a patient's smoking status\n",
    "* extract family history information for genetic studies\n",
    "* redacting patient protected health information from text for research\n",
    "* parsing an excel spreadsheet to generate population statistics about childhood asthma\n",
    "\n",
    "## For this lesson, we will complete a high-level overview of regexs primarily for the purpose of developing with pyConText.\n",
    "\n",
    "For more detailed documentation and explanations, please see the <a href=\"https://docs.python.org/3/library/re.html\">Python regex documentation</a>\n",
    "\n",
    "Let's get started - we will start by importing the re module from the python package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from IPython.display import display, Math, Latex\n",
    "from IPython.display import YouTubeVideo\n",
    "\n",
    "# let's also do a quiz\n",
    "from regular_expression_quiz import which_a_string_not_matched\n",
    "from regular_expression_quiz import test_infiltrates_expression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, before we begin, we will review the various ways regex expressions can be applied. The general approach to using regexs is 1) to compile the search pattern and then 2) to apply the compiled regex to a segment of text:\n",
    "\n",
    "$$complied\\_expression = re.compile(pattern)$$\n",
    "$$extracted\\_text = complied\\_expression.match(string)$$\n",
    "\n",
    "However, if we want to return all instances of matched text found in our target string, we can simplify this two-step process into one-step as follows:\n",
    "\n",
    "$$matches = re.findall(pattern, string)$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying regexs\n",
    "\n",
    "There are several ways a regex pattern so we'll show the most common here and then the rest are listed near the end of this notebook:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- findall: returns all matches of the expression from the string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cardiovascular', 'cardiovascular']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('cardiovascular', \"cardiovascular: patient has cardiovascular disease.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple expressions\n",
    "The simplest regular expressions are very easy to understand.  They will just match any sequence of characters that match the pattern that you provide.  Let's do a few examples of that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['and', 'and']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('and', \"We walked and walked and walked.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['walked', 'walked', 'walked']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('walked', \"We walked and walked and walked.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case sensititivity\n",
    "Note that regular expressions are very explicit in what they match.  This includes character case sensivity (i.e. lowercase and upper case characters.\n",
    "\n",
    "Given this, will the expression below find a match?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('pneumonia', \"Pneumonia reports were inconclusive\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NOTE : pyConText does ignore case by default, but it's important to be aware of case sensitivity for many text applications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequencing ranges\n",
    "How do we get around case sensitivity?  One way is to use character ranges.  In a range, characters can be listed individually in a range like this : \n",
    "\n",
    "* [pP] = Lowercase p or Uppercase P\n",
    "* [abcde]  = Lowercase a, b, c, d, or e\n",
    "\n",
    "Alternatively a range of characters be specified like this : \n",
    "\n",
    "* [a-e] = Any character between lowercase a and lowercase e (this is equivalent to the example above\n",
    "\n",
    "*Sequencing range* additional examples include:\n",
    "- [0-9] = any number between 0 and 9 e.g, 0,1,2,3,4,5..9\n",
    "- [a-z] = any lower case letter between a and z e.g., a,b,c,d...z\n",
    "- [A-Z] = any capitial case letter between A and Z e.g., A, B, ..C.\n",
    "- [A-Za-z0-9] = any alphanumeric regardless of case e.g., A, a, B, b,.. 0, 1..\n",
    "\n",
    "Let's look at our pneumonia capitalization issue again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# : Will this work?\n",
    "re.findall('pneumonia', \"Pneumonia reports were inconclusive\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Pneumonia']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# : What about this one\n",
    "re.findall('[pP]neumonia', \"Pneumonia reports were inconclusive\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's look at one more example.  We'll scan for a range of uppercase letters.\n",
    "Will we match the letter F in this sentence?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A', 'B', 'D']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grades_text = 'Tom got an A.  Sally got a B.  Ralph got a D.  Kelly got an F.'\n",
    "re.findall('[A-E]', grades_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A', 'B', 'D', 'F']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What about this one?\n",
    "re.findall('[A-F]', grades_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A', 'B', 'D', 'F']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And this one\n",
    "re.findall('[ABCDEF]', grades_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Character classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regular expressions represent the strings text that you would like to included and excluded when extracting a string representing a concept. For instance, in the sentence below, we can 1) make a regular expression that only identifies numbers using the *regular expression character class* string \"\\d\", which means one digit 2) compile the expression, and 3) apply the expression to the sentence text.\n",
    "\n",
    "There are a handful of *regular expression character class* including:\n",
    "\n",
    "* \\d   =  One digit -- equivalent to the set [0-9]\n",
    "* \\D   =  One non-digit \n",
    "* \\s   =  One whitespace -- equivalent to the set [ \\t\\n\\r\\f\\v] (i.e. space, tab, newline, carriage return, form feed, vertical tab)\n",
    "* \\S   =  One non-white space\n",
    "* \\w   =  One word character -- equivalent to [a-zA-Z0-9_]\n",
    "* \\W   =  One non-word character\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['9', '1', '2']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('\\d', \"patient is a 91-year old male with 2 broken fingers.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conversely we can generate and apply a *regular expression character class* string that identifies everything, but the numbers by using the string \"\\D\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Coding exercise\n",
    "\n",
    "Create and apply the \\w+ and \\W+ to the text above. What do you see?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['patient',\n",
       " 'is',\n",
       " 'a',\n",
       " '9',\n",
       " 'year',\n",
       " 'old',\n",
       " 'male',\n",
       " 'with',\n",
       " '2',\n",
       " 'broken',\n",
       " 'fingers']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('\\w+', \"patient is a 9-year old male with 2 broken fingers.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Useful character class : Word boundary\n",
    "Sometimes it is very useful working in text to match a certain characters both no more.  For example if a regular expression was written as :\n",
    "\n",
    "* 'cat'\n",
    "\n",
    "This would match the substring 'cat' when found in all of the following words:\n",
    "* cat\n",
    "* category\n",
    "* categories\n",
    "* scatter\n",
    "* etc...\n",
    "\n",
    "For this purpose, the **word boundary** character class exists:\n",
    "\n",
    "* \\b\n",
    "\n",
    "From the Python documentation:\n",
    "\n",
    "Matches the empty string, but only at the beginning or end of a word. A word is defined as a sequence of Unicode alphanumeric or underscore characters, so the end of a word is indicated by whitespace or a non-alphanumeric, non-underscore Unicode character. Note that formally, \\b is defined as the boundary between a \\w and a \\W character (or vice versa), or between \\w and the beginning/end of the string. This means that r'\\bfoo\\b' matches 'foo', 'foo.', '(foo)', 'bar foo baz' but not 'foobar' or 'foo3'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's say that we want to match only 'cat' and none of the other examples above.  Let's see what would happen without word boundaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here are all of the matches on the first expression:\n",
      "['cat', 'cat', 'cat', 'cat']\n"
     ]
    }
   ],
   "source": [
    "# let's try an experiment:\n",
    "cat_text = 'My cat likes to put each category into multiple categories until everyone scatters!'\n",
    "\n",
    "cat_expression_1 = r'cat'\n",
    "cat_matches_1 = re.findall(cat_expression_1, cat_text)\n",
    "print('Here are all of the matches on the first expression:')\n",
    "print(cat_matches_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pro Tip : Python raw strings and character conflicts in Expressions\n",
    "As we move on, we are about to use the **word boundary** character class \\b.  This means something specific in regular expressions, but this sequence also represents the character for a backspace in Python.  There are many special characters (i.e. escape sequences) like this in Python so one way to avoid them is to use a \"raw\" string like this :\n",
    "\n",
    "r'I am a raw string'\n",
    "\n",
    "So that if you use a regular expressions syntax that happens to be the same as a special character (\\b) it can be evaluated as the literal regular expression instead of being convered to using a backspace.  See how we do this below with :\n",
    "\n",
    "r'\\bcat\\b'\n",
    "\n",
    "To see a full list of Python escape sequences, you can <a href=\"https://docs.python.org/3.5/reference/lexical_analysis.html#string-and-bytes-literals\">check them here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We got 4 matches using the expression above and we only wanted one.  Let's see if we can fix that with the word boundary class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here are all of the matches on the second updated expression:\n",
      "['cat']\n"
     ]
    }
   ],
   "source": [
    "cat_expression_2 = r'\\bcat\\b'\n",
    "cat_matches_2 = re.findall(cat_expression_2, cat_text)\n",
    "print('Here are all of the matches on the second updated expression:')\n",
    "print(cat_matches_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What would happen without the \"raw\" string r''?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here are all of the matches on the second updated expression:\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "cat_expression_3 = '\\bcat\\b'\n",
    "cat_matches_3 = re.findall(cat_expression_3, cat_text)\n",
    "print('Here are all of the matches on the second updated expression:')\n",
    "print(cat_matches_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"OR\" matching\n",
    "Regular expresions are powerful since there are special characters that allow for matching multiple sequences or regular expressions at once:\n",
    "\n",
    "* | = match either regular expression on either side of the | symbol  \n",
    "\n",
    "From the Python documentation linked above : 'A|B, where A and B can be arbitrary REs, creates a regular expression that will match either A or B. An arbitrary number of REs can be separated by the '|' in this way'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['blue', 'green', 'red', 'purple']\n",
      "yellow was not matched in this case\n"
     ]
    }
   ],
   "source": [
    "color_text = 'i love the colors blue, green, red and purple but I do not like orange.'\n",
    "\n",
    "color_expression = '(blue|green|red|purple)'\n",
    "color_matches = re.findall(color_expression, color_text)\n",
    "\n",
    "print(color_matches)\n",
    "\n",
    "if 'yellow' in color_matches:\n",
    "    print('yellow was matched in this case')\n",
    "else:\n",
    "    print('yellow was not matched in this case')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quiz : \"OR\" matching\n",
    "\n",
    "Change the regular expression below so that 'hair' is also part of the match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Anatomy matches : ['heart', 'head', 'nose', 'mouth']\n",
      "INCORRECT! hair was not matched by your regular expression.  Please try again.\n"
     ]
    }
   ],
   "source": [
    "anatomy_text = 'Patient physical showed that they had heart, head, nose, mouth and hair'\n",
    "\n",
    "# update the regular expression below so that 'hair' is matched by the code below\n",
    "anatomy_expression = '(heart|head|nose|mouth|UPDATE_ME)'\n",
    "\n",
    "anatomy_matches = re.findall(anatomy_expression, anatomy_text)\n",
    "\n",
    "print('Anatomy matches : ' + str(anatomy_matches))\n",
    "\n",
    "if len(set(['heart', 'head', 'nose', 'mouth']).intersection(set(anatomy_matches))) != 4:\n",
    "    print('INCORRECT! Some of the strings previously matched are no longer matching.  Make sure that heart, head, nose and mouth can all still match!')\n",
    "elif 'hair' not in anatomy_matches:\n",
    "    print('INCORRECT! hair was not matched by your regular expression.  Please try again.')\n",
    "else:\n",
    "    print('CORRECT! hair was matched.  Great job!')\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequencing range and quantifier patterns \n",
    "\n",
    "A *quanitifier* indicates how many of these characters within the range. \n",
    "\n",
    "*Quantifier* simple examples include:\n",
    "- \\*\t = 0 or more\n",
    "- \\+\t = 1 or more\n",
    "- ?\t = 0 or 1\n",
    "\n",
    "Let's create a regular expression using a sequencing range and a quantifier above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['fevers', 'feverish', 'fevers', 'fever']\n"
     ]
    }
   ],
   "source": [
    "txt=\"Patient states fevers she felt feverish before developing many fevers, but no longer has a fever.\"\n",
    "\n",
    "# this will return a list of matches\n",
    "fever_matches = re.findall('fever\\w*', txt)\n",
    "\n",
    "# let's print out all the matches\n",
    "print(fever_matches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['patient',\n",
       " 'is',\n",
       " 'a',\n",
       " '9',\n",
       " 'year',\n",
       " 'old',\n",
       " 'male',\n",
       " 'with',\n",
       " '2',\n",
       " 'broken',\n",
       " 'fingers']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('\\w+', \"patient is a 9-year old male with 2 broken fingers.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz : Quantifiers\n",
    "\n",
    "## Update the function below to pass in the string which would NOT be matched by this regular expression:\n",
    "## 'ah+'\n",
    "\n",
    "* a\n",
    "* ah\n",
    "* ahh\n",
    "* ahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'There should be at least one character \"a\" in your answer since the regular expression expects one \"a\"'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "which_a_string_not_matched('PUT_YOUR_ANSWER_HERE')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz : Quantifiers and Plurals\n",
    "\n",
    "## Update the function below to pass in a regular expression that will match the following strings:\n",
    "\n",
    "* infiltrate\n",
    "* infiltrates\n",
    "\n",
    "## But not match:\n",
    "* infiltratess\n",
    "\n",
    "## HINT : You may want to use the \"?\" special character and also consider the \"\\b\" character class to enforce word boundaries on a match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'INCORRECT.  Your expression matched an unexpected string : [infiltratess].  Please try again.'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_infiltrates_expression(r'infiltrate[s]?')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Back to our initial motivating example...  Above we listed this example which is relevant to our pneumonia work:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$${\\bf infiltrat}e$$\n",
    "$${\\bf infiltrat}ed$$\n",
    "$${\\bf infiltrat}ion$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So how can we match this?  There are a number of ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "infiltrate_text = 'The infiltrate then infiltrated the thing resulting infiltration'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "infiltrate_expression_1 = r'\\b(infiltrate|infiltrated|infiltration)\\b'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "infiltrate_expression_2 = r'\\binfiltrat(e|ed|ion)\\b'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['infiltrate', 'infiltrated', 'infiltration']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(infiltrate_expression_1, infiltrate_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Since the second regular expression has Regular Expression \"groups\" with parentheses, a call to findall() would show us each captured group. Instead, let's iterate and show the spans.  Thisis actually the regular expression interface that pyConText uses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "infiltrate\n",
      "infiltrated\n",
      "infiltration\n"
     ]
    }
   ],
   "source": [
    "infiltrate_compiled = re.compile(infiltrate_expression_2)\n",
    "infiltrate_iter = infiltrate_compiled.finditer(infiltrate_text)\n",
    "\n",
    "for i in infiltrate_iter:\n",
    "    print(i.group())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (OPTIONAL) What are the pros and cons to the two regular expressions above?  Which one do you find to be more readable?  Which would be easier to maintain?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/stopsign.png\">\n",
    "\n",
    "# STOP!  Regular expressions are a complex and deep topic and we don't have time to cover everything.  \n",
    "\n",
    "## For now we've covered most of what will be needed for this course and most cases in pyConText\n",
    "\n",
    "## Let's stop here for now but please come back to this notebook later in the course or sometime after to learn more about more advanced topics in regular expressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional ways to apply Regular Expressions in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- search: checks for a match anywhere in the string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.search(\"and\", \"head and eyes and ears and nose and throat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- match: checks for a match only at the beginning of the string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.match('cardiovascular', \"patient has cardiovascular disease.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- finditer: returns all matches of the expression from the string in an iterative fashion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for extracted in re.finditer('pain(?= in)', \"pain on the neck; pain in the jaw\"):\n",
    "    print(extracted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Quantifiers\n",
    "*Quantifier* examples also include:\n",
    "- {5} = 5 exactly\n",
    "- {5,7} = between 5 and 7\n",
    "- {1, } = 1 or more\n",
    "- {, 4} = up to 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Positional patterns\n",
    "\n",
    "For some expressions the position of the string is an important aspect to capture. \n",
    "\n",
    "*Positional* examples:\n",
    "- ^ Match the start of the string\n",
    "- $ Match the end of the string\n",
    "\n",
    "For example, in the sentence below, we may only be interested in identifying the headers from the text strings. How can we apply the *positional expression* to capture the headers?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt=\"cardiovascular: patient has cardiovascular\"\n",
    "re.search('cardiovascular$', txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtaining span offsets\n",
    "\n",
    "The expression found the string 'cardiovascular' at the start of the sentence rather than near the end. We can not only find strings, but we can verify this claim and obtain information about the string identified using match, which will give us the span and the matched string below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matched=re.match('^cardiovascular', txt)\n",
    "print(matched)\n",
    "print(\"match:\", matched.group())\n",
    "print(\"span:\", matched.start(),matched.end())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Identifying the start and end of a string can be important for many reasons:\n",
    "* marking up text with denoting information of interest using html or xml\n",
    "* redacting sensitive information from text\n",
    "\n",
    "Below we demonstrate how to apply the matched string from the pattern back onto the original text by splicing the string according to the start and end of the matched pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt[matched.start():matched.end()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Extracting a set of strings\n",
    "\n",
    "In some cases, we may want to extract a group of strings that match a more complex pattern.\n",
    "To do this we will use a pattern with the following syntax: $$(?P<name>...)$$ \n",
    "\n",
    "Note you can retrieve the individual span parts using the 'name' or position in the retrieved tuple. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt=\"admission date:6/15-2015.\"\n",
    "fullSpan=re.match(r\"admission date:\\s*(?P<month>\\d{1,2})[-|\\/](?P<day>\\d{1,2})-(?P<year>\\d{2,4})\", txt)\n",
    "print(fullSpan, \"\\n\")\n",
    "\n",
    "print(\"retrieved in as a tuple/ full span\")\n",
    "print(\"all parts:\", fullSpan.groups(), \"=\", fullSpan.group(0), \"\\n\")\n",
    "\n",
    "print(\"retrieved as a dictionary\")\n",
    "print(fullSpan.groupdict(),\"\\n\")\n",
    "\n",
    "print(\"retrieved in parts\")\n",
    "print(\"month:\",fullSpan.group('month'), \"=\", fullSpan.group(1) )\n",
    "print(\"day:\",fullSpan.group('day'), \"=\", fullSpan.group(2))\n",
    "print(\"year:\",fullSpan.group('year'), \"=\", fullSpan.group(3))\n",
    "print(fullSpan.groupdict())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Coding exercise\n",
    "\n",
    "Create an expression using the group syntax above to identify the patient name (Jane Doe) of the following text. Then replace it with the new name (Mary Lamb).\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt=\"Patient Name: Jane Doe\"\n",
    "re.subn(\"(?<=Patient Name: )\\w+ \\w+\", \"Mary Lamb\", txt)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For more detailed information about regexs\n",
    "\n",
    "Please view the following video by Professors Dan Jurafsky & Chris Manning\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "YouTubeVideo(\"hwDhO1GLb_4?list=PL6397E4B26D00A269\", width = 800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Want to try some expressions without coding?\n",
    "Try [pythex](http://pythex.org/) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/><br/>This material presented as part of the DeCART Data Science for the Health Science Summer Program at the University of Utah in 2019.<br/>\n",
    "Presenters : Dr. Wendy Chapman, Kelly Peterson, Alec Chapman, Jianlin Shi <br> Acknowledgement: Many thanks to Olga Patterson because part of the materials are adopted from his previous work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "widgets": {
   "state": {},
   "version": "1.1.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}