Appendix_text_processing_txtblob.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Note : This notebook was prepared for a previous NLP course by Brian Chapman, Wendy Chapman, Jianlin Shi and Danielle Mowery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clinical natural language processing\n",
    "\n",
    "Clinical texts contain a summary of the patient's clinical story and their physiological status changes over time. Specifically, members of the care providing team document these changes by describing the patient's status upon admission, during examination, throughout their treatment, and upon discharge/transfer. What kinds of secondary use (beyond patient care) could be supported from information extracted from textual data e.g., clinical reports?\n",
    "\n",
    "* Recruit patients for clinical trials\n",
    "* Enrich structured problem lists with missed problems\n",
    "* Build cohorts for comparative effectiveness studies\n",
    "* Extract a patient's family history information for assessing predisposition of disease\n",
    "\n",
    "\n",
    "Below is a sample report from <a href=http://www.mtsamples.com/site/pages/sample.asp?Type=91-SOAP%20/%20Chart%20/%20Progress%20Notes&Sample=40-Cardiology%20Progress%20Note>\"MTSamples.com\"</a>\n",
    "\n",
    "![MTSamples.com](./images/MT_Samples_report.png)\n",
    "\n",
    "From the report above, how would you tell me whether the patient has had chest pain? What information did you use to induce that?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Characterizing textual reports\n",
    "\n",
    "What information is described in the report above?\n",
    "* symptoms\n",
    "* signs\n",
    "* tests\n",
    "* treatments\n",
    "\n",
    "What type of structures are used to construct the report?\n",
    "* words\n",
    "* phrases\n",
    "* sentences\n",
    "* sections\n",
    "* SOAP structure\n",
    "\n",
    "How does this report structure differ from other texts e.g., newspaper articles? How is it similar? \n",
    "\n",
    "In this lesson, we are going to learn how to analyze information within reports that can be leveraged to distinguish report types from each other (discharge summary vs radiology reports), classify patients (diseased vs healthy), and other use cases.\n",
    "\n",
    "\n",
    "# Reading in files\n",
    "\n",
    "Again, one of the fundamental steps to processing and analyzing textual reports, is locating files to read into memory. To get started we are going to call in or 'import' some tools that allow us to read in files from the computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents\n"
     ]
    }
   ],
   "source": [
    "import glob, os\n",
    "print(os.getcwd()+r\"/data/mtsamples_documents\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Reading in texts from the directory\n",
    "\n",
    "Next, we want to read in a set of text files using glob to begin processing them. Glob will return us a list of text files with their corresponding paths.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/289-Dermatitis - SOAP.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1602-SOAP - Shortness of Breath.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/73-Gen Med SOAP - 3.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1601-SOAP - Lung Mass.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1439-SOAP - Substance Abuse.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/450-URI - SOAP.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/452-Fifth Disease - SOAP 2.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1198-SOAP - Temporal Mass 2.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/288-Acne - SOAP 2.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/37-Gen Med SOAP - 2.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/2628-Atrial Fibrillation - SOAP.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1438-SOAP - Numbness & Tingling.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1134-Diabetes Mellitus - SOAP Note - 2.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/378-Gen Med SOAP - 7.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/406-Poison Ivy - SOAP.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/72-Gen Med SOAP - 1.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1582-Gen Med SOAP - 11.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1440-Gen Med SOAP.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/227-Diabetes Mellitus - SOAP Note - 1.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1441-Gen Med SOAP - 9.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/74-Gen Med SOAP - 4.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/407-Wasp Sting - SOAP 2.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1630-SOAP - Cholecystitis.txt', '/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/2627-COPD & Pneumonia - SOAP.txt']\n"
     ]
    }
   ],
   "source": [
    "files=glob.glob(os.getcwd()+\"/data/mtsamples_documents/*.txt\")\n",
    "print(files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Selecting a file for processing\n",
    "\n",
    "Using the list and indexing, we can select a file containing a progress note report to process. Use the index to select a report of interest to you. We will select the seventh file in the list from the end. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/jianlins/work/decart_rule_based_nlp/data/mtsamples_documents/1440-Gen Med SOAP.txt'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files[-7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Opening a file for processing\n",
    "\n",
    "Now that we've selected a file to process, we can use 'open' and 'read' to read all the file contents (report) into memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample Name: Gen Med SOAP  Description:  Palpitations, possibly related to anxiety. Fatigue. Loose stools with some green color and also some nausea.\n",
      "(Medical Transcription Sample Report)\n",
      "SUBJECTIVE:  This patient presents to the office today because he has not been feeling well.  He was in for a complete physical on 05/02/2008.  According to the chart, the patient gives a history of feeling bad for about two weeks.  At first he thought it was stress and anxiety and then he became worried it was something else.  He says he is having a lot of palpitations.  He gets a fluttering feeling in his chest.  He has been very tired over two weeks as well.  His job has been really getting to him.  He has been feeling nervous and anxious.  It seems like when he is feeling stressed he has more palpitations, sometimes they cause chest pain.  These symptoms are not triggered by exertion.  He had similar symptoms about 9 or 10 years ago.  At that time he went through a full workup.  Everything ended up being negative and they gave him something that he took for his nerves and he says that helped.  Unfortunately, he does not remember what it was.  Also over the last three days he has had some intestinal problems.  He has had some intermittent nausea and his stools have been loose.  He has been having some really funny green color to his bowel movements.  There has been no blood in the stool.  He is not having any abdominal pain, just some nausea.  He does not have much of an appetite.  He is a nonsmoker.\n",
      "\n",
      "OBJECTIVE:  His weight today is 168.4 pounds, blood pressure 142/76, temperature 97.7, pulse 68, and respirations 16.  General exam:  The patient is nontoxic and in no acute distress.  There is no labored breathing.  Psychiatric:  He is alert and oriented times 3.  Ears:  Tympanic membranes pearly gray bilaterally.  Mouth:  No erythema, ulcers, vesicles, or exudate noted.  Eyes:  Pupils equal, round, and reactive to light bilaterally.  Neck is supple.  No lymphadenopathy.  Lungs:  Clear to auscultation.  No rales, rhonchi, or wheezing.  Cardiac:  Regular rate and rhythm without murmur.  Extremities:  No edema, cyanosis, or clubbing.\n",
      "\n",
      "ASSESSMENT:  \n",
      "1.  Palpitations, possibly related to anxiety.\n",
      "2.  Fatigue.\n",
      "3.  Loose stools with some green color and also some nausea.  There has been no vomiting, possibly a touch of gastroenteritis going on here.\n",
      "\n",
      "PLAN:  The patient admits he has been putting this off now for about two weeks.  He says his work is definitely contributing to some of his symptoms and he feels stressed.  He is leaving for a vacation very soon.  Unfortunately, he is actually leaving Wednesday for XYZ, which puts us into a bit of a bind in terms of doing testing on him.  My overall opinion is he has some anxiety related issues and he may also have a touch of gastroenteritis.  A 12-lead EKG was performed on him in the office today.  This EKG was compared with the previous EKG contained in the chart from 2006 and I see that these EKGs look very similar with no significant changes noted, which is definitely a good news.  I am going to send him to the lab from our office to get the following tests done:  Comprehensive metabolic profile, CBC, urinalysis with reflex to culture and we will also get a chest X-ray.  Tomorrow morning I will manage to schedule him for an exercise stress test at Bad Axe Hospital.  We were able to squeeze him in.  His appointment is at 8:15 in the morning.  He is going to have the stress test done in the morning and he will come back to the office in the afternoon for recheck.  I am not going to be here so he is going to see Dr. X.  Dr. X should hopefully be able to call over and speak with the physician who attended the stress test and get a preliminary result before he leaves for XYZ.  Certainly, if something comes up we may need to postpone his trip.  We petitioned his medical records from his former physician and with luck we will be able to find out what medication he was on about nine or ten years ago.  In the meantime I have given him Ativan 0.5 mg one tablet two to three times a day as needed for anxiety.  I talked about Ativan, how it works.  I talked about the side effects.  I told him to use it only as needed and we can see how he is doing tomorrow when he comes back for his recheck.  I took him off of work today and tomorrow so he could rest.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "report_text=open(files[-7]).read()\n",
    "print(report_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Segmenting a report \n",
    "\n",
    "One of the fundamental tasks to text processing is segmenting the text into more managable parts e.g., section segmentation or sentence segmentation. For many NLP tasks such as information extraction, a single sentence can contain most information about an entity or event. Using Textblob, we can segment the report into sentences. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample Name: Gen Med SOAP  Description:  Palpitations, possibly related to anxiety.\n",
      "-----------------------\n",
      "Fatigue.\n",
      "-----------------------\n",
      "Loose stools with some green color and also some nausea.\n",
      "-----------------------\n",
      "(Medical Transcription Sample Report)\n",
      "SUBJECTIVE:  This patient presents to the office today because he has not been feeling well.\n",
      "-----------------------\n",
      "He was in for a complete physical on 05/02/2008.\n",
      "-----------------------\n",
      "According to the chart, the patient gives a history of feeling bad for about two weeks.\n",
      "-----------------------\n",
      "At first he thought it was stress and anxiety and then he became worried it was something else.\n",
      "-----------------------\n",
      "He says he is having a lot of palpitations.\n",
      "-----------------------\n",
      "He gets a fluttering feeling in his chest.\n",
      "-----------------------\n",
      "He has been very tired over two weeks as well.\n",
      "-----------------------\n",
      "His job has been really getting to him.\n",
      "-----------------------\n",
      "He has been feeling nervous and anxious.\n",
      "-----------------------\n",
      "It seems like when he is feeling stressed he has more palpitations, sometimes they cause chest pain.\n",
      "-----------------------\n",
      "These symptoms are not triggered by exertion.\n",
      "-----------------------\n",
      "He had similar symptoms about 9 or 10 years ago.\n",
      "-----------------------\n",
      "At that time he went through a full workup.\n",
      "-----------------------\n",
      "Everything ended up being negative and they gave him something that he took for his nerves and he says that helped.\n",
      "-----------------------\n",
      "Unfortunately, he does not remember what it was.\n",
      "-----------------------\n",
      "Also over the last three days he has had some intestinal problems.\n",
      "-----------------------\n",
      "He has had some intermittent nausea and his stools have been loose.\n",
      "-----------------------\n",
      "He has been having some really funny green color to his bowel movements.\n",
      "-----------------------\n",
      "There has been no blood in the stool.\n",
      "-----------------------\n",
      "He is not having any abdominal pain, just some nausea.\n",
      "-----------------------\n",
      "He does not have much of an appetite.\n",
      "-----------------------\n",
      "He is a nonsmoker.\n",
      "-----------------------\n",
      "OBJECTIVE:  His weight today is 168.4 pounds, blood pressure 142/76, temperature 97.7, pulse 68, and respirations 16.\n",
      "-----------------------\n",
      "General exam:  The patient is nontoxic and in no acute distress.\n",
      "-----------------------\n",
      "There is no labored breathing.\n",
      "-----------------------\n",
      "Psychiatric:  He is alert and oriented times 3.\n",
      "-----------------------\n",
      "Ears:  Tympanic membranes pearly gray bilaterally.\n",
      "-----------------------\n",
      "Mouth:  No erythema, ulcers, vesicles, or exudate noted.\n",
      "-----------------------\n",
      "Eyes:  Pupils equal, round, and reactive to light bilaterally.\n",
      "-----------------------\n",
      "Neck is supple.\n",
      "-----------------------\n",
      "No lymphadenopathy.\n",
      "-----------------------\n",
      "Lungs:  Clear to auscultation.\n",
      "-----------------------\n",
      "No rales, rhonchi, or wheezing.\n",
      "-----------------------\n",
      "Cardiac:  Regular rate and rhythm without murmur.\n",
      "-----------------------\n",
      "Extremities:  No edema, cyanosis, or clubbing.\n",
      "-----------------------\n",
      "ASSESSMENT:  \n",
      "1.\n",
      "-----------------------\n",
      "Palpitations, possibly related to anxiety.\n",
      "-----------------------\n",
      "2.\n",
      "-----------------------\n",
      "Fatigue.\n",
      "-----------------------\n",
      "3.\n",
      "-----------------------\n",
      "Loose stools with some green color and also some nausea.\n",
      "-----------------------\n",
      "There has been no vomiting, possibly a touch of gastroenteritis going on here.\n",
      "-----------------------\n",
      "PLAN:  The patient admits he has been putting this off now for about two weeks.\n",
      "-----------------------\n",
      "He says his work is definitely contributing to some of his symptoms and he feels stressed.\n",
      "-----------------------\n",
      "He is leaving for a vacation very soon.\n",
      "-----------------------\n",
      "Unfortunately, he is actually leaving Wednesday for XYZ, which puts us into a bit of a bind in terms of doing testing on him.\n",
      "-----------------------\n",
      "My overall opinion is he has some anxiety related issues and he may also have a touch of gastroenteritis.\n",
      "-----------------------\n",
      "A 12-lead EKG was performed on him in the office today.\n",
      "-----------------------\n",
      "This EKG was compared with the previous EKG contained in the chart from 2006 and I see that these EKGs look very similar with no significant changes noted, which is definitely a good news.\n",
      "-----------------------\n",
      "I am going to send him to the lab from our office to get the following tests done:  Comprehensive metabolic profile, CBC, urinalysis with reflex to culture and we will also get a chest X-ray.\n",
      "-----------------------\n",
      "Tomorrow morning I will manage to schedule him for an exercise stress test at Bad Axe Hospital.\n",
      "-----------------------\n",
      "We were able to squeeze him in.\n",
      "-----------------------\n",
      "His appointment is at 8:15 in the morning.\n",
      "-----------------------\n",
      "He is going to have the stress test done in the morning and he will come back to the office in the afternoon for recheck.\n",
      "-----------------------\n",
      "I am not going to be here so he is going to see Dr. X.  Dr. X should hopefully be able to call over and speak with the physician who attended the stress test and get a preliminary result before he leaves for XYZ.\n",
      "-----------------------\n",
      "Certainly, if something comes up we may need to postpone his trip.\n",
      "-----------------------\n",
      "We petitioned his medical records from his former physician and with luck we will be able to find out what medication he was on about nine or ten years ago.\n",
      "-----------------------\n",
      "In the meantime I have given him Ativan 0.5 mg one tablet two to three times a day as needed for anxiety.\n",
      "-----------------------\n",
      "I talked about Ativan, how it works.\n",
      "-----------------------\n",
      "I talked about the side effects.\n",
      "-----------------------\n",
      "I told him to use it only as needed and we can see how he is doing tomorrow when he comes back for his recheck.\n",
      "-----------------------\n",
      "I took him off of work today and tomorrow so he could rest.\n",
      "-----------------------\n"
     ]
    }
   ],
   "source": [
    "from textblob import TextBlob\n",
    "\n",
    "#sentence object\n",
    "blob = TextBlob(report_text)\n",
    "\n",
    "for sent in blob.sentences:\n",
    "    print(sent)\n",
    "    print(\"-----------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Characterizing sentences\n",
    "\n",
    "Because Textblob returns a list, we can also count the number of sentences in the report. How many sentences are there in the report? How might knowing the number of sentences be important to guessing a report type? \n",
    "\n",
    "We can also obtain a particular sentence in the list using the index. What does the 10th sentence read? When could it be important to select a sentence by its index?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65\n",
      "His job has been really getting to him.\n"
     ]
    }
   ],
   "source": [
    "print(len(blob.sentences))\n",
    "\n",
    "print(blob.sentences[10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenizing sentences into words\n",
    "\n",
    "Words are the basic building blocks of text. As humans, when we want to find relevant information from text we search text for words in sentences of the report related to the topic of interest to us. Textblob supports tokenization by providing the words or tokens from the report text. What's the difference between these two functions?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Sample', 'Name', 'Gen', 'Med', 'SOAP', 'Description', 'Palpitations', 'possibly', 'related', 'to', 'anxiety', 'Fatigue', 'Loose', 'stools', 'with', 'some', 'green', 'color', 'and', 'also', 'some', 'nausea', 'Medical', 'Transcription', 'Sample', 'Report', 'SUBJECTIVE', 'This', 'patient', 'presents', 'to', 'the', 'office', 'today', 'because', 'he', 'has', 'not', 'been', 'feeling', 'well', 'He', 'was', 'in', 'for', 'a', 'complete', 'physical', 'on', '05/02/2008']\n",
      "=================\n",
      "['Sample', 'Name', ':', 'Gen', 'Med', 'SOAP', 'Description', ':', 'Palpitations', ',', 'possibly', 'related', 'to', 'anxiety', '.', 'Fatigue', '.', 'Loose', 'stools', 'with', 'some', 'green', 'color', 'and', 'also', 'some', 'nausea', '.', '(', 'Medical', 'Transcription', 'Sample', 'Report', ')', 'SUBJECTIVE', ':', 'This', 'patient', 'presents', 'to', 'the', 'office', 'today', 'because', 'he', 'has', 'not', 'been', 'feeling', 'well']\n"
     ]
    }
   ],
   "source": [
    "print(blob.words[0:50])\n",
    "\n",
    "print(\"=================\")\n",
    "\n",
    "print(blob.tokens[0:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Counting words in the text\n",
    "\n",
    "In some cases, knowing the frequency of words in text can help us better understand the content of the text and classify it by topic. For example, if we know a report contains many terms associated with the heart e.g., cardiology, echocardiogram, heart, infarction, we may deduce that the patient may have been seen for cardiovascular problems. If we review patient satisfaction surveys with many positive terms, we may deduce that the patient had a satisfactory visit. Textblob can give us word counts from a report text. Which words are most frequent? Are they always informative? HINT: How frequent is the word 'and'?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "defaultdict(<class 'int'>, {'sample': 2, 'name': 1, 'gen': 1, 'med': 1, 'soap': 1, 'description': 1, 'palpitations': 4, 'possibly': 3, 'related': 3, 'to': 24, 'anxiety': 5, 'fatigue': 2, 'loose': 3, 'stools': 3, 'with': 7, 'some': 10, 'green': 3, 'color': 3, 'and': 23, 'also': 5, 'nausea': 4, 'medical': 2, 'transcription': 1, 'report': 1, 'subjective': 1, 'this': 3, 'patient': 4, 'presents': 1, 'the': 21, 'office': 4, 'today': 4, 'because': 1, 'he': 38, 'has': 12, 'not': 6, 'been': 9, 'feeling': 5, 'well': 2, 'was': 7, 'in': 12, 'for': 11, 'a': 16, 'complete': 1, 'physical': 1, 'on': 5, '05/02/2008': 1, 'according': 1, 'chart': 2, 'gives': 1, 'history': 1, 'of': 9, 'bad': 2, 'about': 6, 'two': 4, 'weeks': 3, 'at': 4, 'first': 1, 'thought': 1, 'it': 6, 'stress': 4, 'then': 1, 'became': 1, 'worried': 1, 'something': 3, 'else': 1, 'says': 3, 'is': 18, 'having': 3, 'lot': 1, 'gets': 1, 'fluttering': 1, 'his': 13, 'chest': 3, 'very': 3, 'tired': 1, 'over': 3, 'as': 3, 'job': 1, 'really': 2, 'getting': 1, 'him': 10, 'nervous': 1, 'anxious': 1, 'seems': 1, 'like': 1, 'when': 2, 'stressed': 2, 'more': 1, 'sometimes': 1, 'they': 2, 'cause': 1, 'pain': 2, 'these': 2, 'symptoms': 3, 'are': 1, 'triggered': 1, 'by': 1, 'exertion': 1, 'had': 3, 'similar': 2, '9': 1, 'or': 5, '10': 1, 'years': 2, 'ago': 2, 'that': 4, 'time': 1, 'went': 1, 'through': 1, 'full': 1, 'workup': 1, 'everything': 1, 'ended': 1, 'up': 2, 'being': 1, 'negative': 1, 'gave': 1, 'took': 2, 'nerves': 1, 'helped': 1, 'unfortunately': 2, 'does': 2, 'remember': 1, 'what': 2, 'last': 1, 'three': 2, 'days': 1, 'intestinal': 1, 'problems': 1, 'intermittent': 1, 'have': 5, 'funny': 1, 'bowel': 1, 'movements': 1, 'there': 3, 'no': 9, 'blood': 2, 'stool': 1, 'any': 1, 'abdominal': 1, 'just': 1, 'much': 1, 'an': 2, 'appetite': 1, 'nonsmoker': 1, 'objective': 1, 'weight': 1, '168.4': 1, 'pounds': 1, 'pressure': 1, '142/76': 1, 'temperature': 1, '97.7': 1, 'pulse': 1, '68': 1, 'respirations': 1, '16': 1, 'general': 1, 'exam': 1, 'nontoxic': 1, 'acute': 1, 'distress': 1, 'labored': 1, 'breathing': 1, 'psychiatric': 1, 'alert': 1, 'oriented': 1, 'times': 2, '3': 2, 'ears': 1, 'tympanic': 1, 'membranes': 1, 'pearly': 1, 'gray': 1, 'bilaterally': 2, 'mouth': 1, 'erythema': 1, 'ulcers': 1, 'vesicles': 1, 'exudate': 1, 'noted': 2, 'eyes': 1, 'pupils': 1, 'equal': 1, 'round': 1, 'reactive': 1, 'light': 1, 'neck': 1, 'supple': 1, 'lymphadenopathy': 1, 'lungs': 1, 'clear': 1, 'auscultation': 1, 'rales': 1, 'rhonchi': 1, 'wheezing': 1, 'cardiac': 1, 'regular': 1, 'rate': 1, 'rhythm': 1, 'without': 1, 'murmur': 1, 'extremities': 1, 'edema': 1, 'cyanosis': 1, 'clubbing': 1, 'assessment': 1, '1': 1, '2': 1, 'vomiting': 1, 'touch': 2, 'gastroenteritis': 2, 'going': 5, 'here': 2, 'plan': 1, 'admits': 1, 'putting': 1, 'off': 2, 'now': 1, 'work': 2, 'definitely': 2, 'contributing': 1, 'feels': 1, 'leaving': 2, 'vacation': 1, 'soon': 1, 'actually': 1, 'wednesday': 1, 'xyz': 2, 'which': 2, 'puts': 1, 'us': 1, 'into': 1, 'bit': 1, 'bind': 1, 'terms': 1, 'doing': 2, 'testing': 1, 'my': 1, 'overall': 1, 'opinion': 1, 'issues': 1, 'may': 2, '12-lead': 1, 'ekg': 3, 'performed': 1, 'compared': 1, 'previous': 1, 'contained': 1, 'from': 3, '2006': 1, 'i': 9, 'see': 3, 'ekgs': 1, 'look': 1, 'significant': 1, 'changes': 1, 'good': 1, 'news': 1, 'am': 2, 'send': 1, 'lab': 1, 'our': 1, 'get': 3, 'following': 1, 'tests': 1, 'done': 2, 'comprehensive': 1, 'metabolic': 1, 'profile': 1, 'cbc': 1, 'urinalysis': 1, 'reflex': 1, 'culture': 1, 'we': 6, 'will': 4, 'x-ray': 1, 'tomorrow': 3, 'morning': 3, 'manage': 1, 'schedule': 1, 'exercise': 1, 'test': 3, 'axe': 1, 'hospital': 1, 'were': 1, 'able': 3, 'squeeze': 1, 'appointment': 1, '8:15': 1, 'come': 1, 'back': 2, 'afternoon': 1, 'recheck': 2, 'be': 3, 'so': 2, 'dr': 2, 'x': 2, 'should': 1, 'hopefully': 1, 'call': 1, 'speak': 1, 'physician': 2, 'who': 1, 'attended': 1, 'preliminary': 1, 'result': 1, 'before': 1, 'leaves': 1, 'certainly': 1, 'if': 1, 'comes': 2, 'need': 1, 'postpone': 1, 'trip': 1, 'petitioned': 1, 'records': 1, 'former': 1, 'luck': 1, 'find': 1, 'out': 1, 'medication': 1, 'nine': 1, 'ten': 1, 'meantime': 1, 'given': 1, 'ativan': 2, '0.5': 1, 'mg': 1, 'one': 1, 'tablet': 1, 'day': 1, 'needed': 2, 'talked': 2, 'how': 2, 'works': 1, 'side': 1, 'effects': 1, 'told': 1, 'use': 1, 'only': 1, 'can': 1, 'could': 1, 'rest': 1})\n",
      "--------------------------------\n",
      "and: 23\n"
     ]
    }
   ],
   "source": [
    "print(blob.word_counts)\n",
    "\n",
    "\n",
    "print(\"--------------------------------\")\n",
    "print(\"and:\",blob.words.count('and'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Coding exercise I\n",
    "\n",
    "What are the 5 most frequent words in the dictionary?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenizing multi-words (n-grams)\n",
    "\n",
    "Signs, symptoms, diagnoses are often made from more than one word. In addition to individual words, we can define windows of words with varying lengths of tokens (1-token = unigrams; 2-token = bigrams, etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(\"Bigrams:\", blob.ngrams(n=2))\n",
    "print(\"Trigram - right upper extremity:\", blob.ngrams(n=3).count(['right', 'upper', 'extremity']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lemmatizing words to their stems (morphology)\n",
    "\n",
    "Words can have many variants based on parts of speech. Textblob supports lemmatizing words to their root form. This can be important for identifying a concept documented with many lexical variants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# before we perform syntactic operations like lemmatization we should make sure that we have some resources available\n",
    "# including WordNet and some taggers\n",
    "import nltk\n",
    "nltk.download(\"wordnet\")\n",
    "nltk.download(\"averaged_perceptron_tagger\")\n",
    "nltk.download(\"brown\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from textblob import Word\n",
    "w = Word(\"smokes\")\n",
    "print(\"lemma of smokes is:\", w.lemmatize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How well does a lemmatizer trained on general English work for medical language?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tagging parts of speech (syntax)\n",
    "\n",
    "Signs, symptoms, diagnoses are often nouns. Textblob can give us the parts of speech of words from the text. There are two sets of tags commonly used for part of speech tagging: [Penn Treebank] (https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html\") and [Brown Corpus tags] (http://www.comp.leeds.ac.uk/ccalas/tagsets/brown.html) Let's briefly review the different standards for tagging parts of speech of text. Which tag set appears below?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(blob.tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parsing noun phrases\n",
    "\n",
    "Not only are signs, symptoms, diagnoses often nouns, but they can be mulit-word and part of noun phrases; therefore, parsing phrases based on part of speech can help us identify words that when grouped together describe a salient clinical concept e.g., 'coronary artery disease' --> ('coronary', 'JJ'), ('artery', 'NN'), ('disease', 'NN') rather than just 'coronary' ('coronary', 'JJ'). What types clinical concepts are most frequently represented by noun phrases in the text? Which seem correct and which don't seem correct?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(blob.np_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Coding exercise III\n",
    "\n",
    "Print the noun phrases that occur more than 1 time in the text above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Viewing noun phrases in text\n",
    "\n",
    "Part of speech tags in conjuction with named entity recognition labels can depict how clinical concepts are created from words create phrases and clauses. Can you identify the beginning (B), inside (I) and outside (O) of each phrase? In addition to BIO labelling there are other labels standards. [Take a look!] (https://lingpipe-blog.com/2009/10/14/coding-chunkers-as-taggers-io-bio-bmewo-and-bmewo/) \n",
    "\n",
    "Go here to experiment with parse trees: foxtype.com/sentence-tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "blob.parse()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Determining meaning of words (semantics)\n",
    "\n",
    "Words, phrases, and clauses convey meaning or the semantics of text by describing entities and events within clinical text. The meaning of a concept can be derived from a lexicon or dictionary of words.\n",
    "For instance, we can define the concept of Addison’s disease with its synonyms (words sharing similar meaning) e.g.,\n",
    "adrenocortical insufficiency. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "vertigo = Word('vertigo')\n",
    "print(\"Vertigo: \",vertigo.definitions)\n",
    "\n",
    "\n",
    "dizziness = Word('dizziness')\n",
    "print(\"Dizziness: \",vertigo.definitions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words have synonyms, and there are many dictionaries that will provide synonyms. WordNet is the most comprehensive but the least focused on medical words (although it has many medical words). The UMLS Metathesaurus is a thesaurus of thesauri that has synonyms for many medical and biological terms.\n",
    "\n",
    "Check out the synonymize function someone wrote: rwet.decontextualize.com/book/textblob/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from textblob.wordnet import NOUN\n",
    "synsets = dizziness.get_synsets(pos=NOUN)\n",
    "print(synsets[0].lemma_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Identifying many senses of a word\n",
    "\n",
    "In addition to one word being similar to another word, one word can convey many meanings (i.e., a word can be polysemous), e.g., discharge can be release of a patient from care or a substance from an abscess. Discerning the sense or meaning of a word is called word sense disambiguation.  Using Textblob and the underlying dictionary <a href=\"https://wordnet.princeton.edu/wordnet/\">WordNet</a>), we can obtain the definition and corresponding synsets (i.e., word senses) of a word. Are any of the synsets for 'admit' below clinical? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from textblob.wordnet import VERB\n",
    "\n",
    "word = Word('admit')\n",
    "for definition, syn in zip(word.definitions, word.get_synsets(pos=VERB)):\n",
    "    print(definition+\" --> \"+str(syn)+\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Determine the degree of similarity between synsets of two words\n",
    "\n",
    "Given two words, we can also compare the synsets to determine the degree of semantic similarity. How similar are the pus and discharge synsets below? Can you think of other words with more semantic similarity? Try them! Test different v. of the same word to look for similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from textblob.wordnet import Synset\n",
    "\n",
    "declare_true=Synset('admit.v.01')\n",
    "give_access=Synset('admit.v.05')\n",
    "print(\"Similarity: \", declare_true.path_similarity(give_access))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Synonyms\n",
    "Use the same code to see how similar two different words are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "declare_true=Synset('dizziness.n.01')\n",
    "give_access=Synset('vertigo.n.01')\n",
    "print(\"Similarity: \", declare_true.path_similarity(give_access))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "widgets": {
   "state": {},
   "version": "1.1.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}