Assignment+2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.0** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2 - Introduction to NLTK\n",
    "\n",
    "In part 1 of this assignment you will use nltk to explore the Herman Melville novel Moby Dick. Then in part 2 you will create a spelling recommender function that uses nltk to find words similar to the misspelling. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Analyzing Moby Dick"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Introductory Examples for the NLTK Book ***\n",
      "Loading text1, ..., text9 and sent1, ..., sent9\n",
      "Type the name of the text or sentence to view it.\n",
      "Type: 'texts()' or 'sents()' to list the materials.\n",
      "text1: Moby Dick by Herman Melville 1851\n",
      "text2: Sense and Sensibility by Jane Austen 1811\n",
      "text3: The Book of Genesis\n",
      "text4: Inaugural Address Corpus\n",
      "text5: Chat Corpus\n",
      "text6: Monty Python and the Holy Grail\n",
      "text7: Wall Street Journal\n",
      "text8: Personals Corpus\n",
      "text9: The Man Who Was Thursday by G . K . Chesterton 1908\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from nltk.book import *\n",
    "\n",
    "# If you would like to work with the raw text you can use 'moby_raw'\n",
    "with open('moby.txt', 'r') as f:\n",
    "    moby_raw = f.read()\n",
    "    \n",
    "# If you would like to work with the novel in nltk.Text format you can use 'text1'\n",
    "moby_tokens = nltk.word_tokenize(moby_raw)\n",
    "text1 = nltk.Text(moby_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1\n",
    "\n",
    "How many tokens (words and punctuation symbols) are in text1?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "254989"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def example_one():\n",
    "    \n",
    "    return len(nltk.word_tokenize(moby_raw)) # or alternatively len(text1)\n",
    "\n",
    "example_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2\n",
    "\n",
    "How many unique tokens (unique words and punctuation) does text1 have?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20755"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def example_two():\n",
    "    \n",
    "    return len(set(nltk.word_tokenize(moby_raw))) # or alternatively len(set(text1))\n",
    "\n",
    "example_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3\n",
    "\n",
    "After lemmatizing the verbs, how many unique tokens does text1 have?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16900"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "def example_three():\n",
    "\n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "    lemmatized = [lemmatizer.lemmatize(w,'v') for w in text1]\n",
    "\n",
    "    return len(set(lemmatized))\n",
    "\n",
    "example_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "\n",
    "What is the lexical diversity of the given text input? (i.e. ratio of unique tokens to the total number of tokens)\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08139566804842562"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_one():\n",
    "    \n",
    "    ratio = len(set(text1))/len(text1)\n",
    "    return ratio\n",
    "\n",
    "answer_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "What percentage of tokens is 'whale'or 'Whale'?\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dist = {}\n",
    "for i in text1:\n",
    "    dist[i] = dist.get(i,0)+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4125668166077752"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_two():\n",
    "    \n",
    "    \n",
    "    per = (dist['whale'] + dist['Whale'])/len(text1) * 100\n",
    "    return per\n",
    "\n",
    "answer_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?\n",
    "\n",
    "*This function should return a list of 20 tuples where each tuple is of the form `(token, frequency)`. The list should be sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(',', 19204),\n",
       " ('the', 13715),\n",
       " ('.', 7308),\n",
       " ('of', 6513),\n",
       " ('and', 6010),\n",
       " ('a', 4545),\n",
       " ('to', 4515),\n",
       " (';', 4173),\n",
       " ('in', 3908),\n",
       " ('that', 2978),\n",
       " ('his', 2459),\n",
       " ('it', 2196),\n",
       " ('I', 2097),\n",
       " ('!', 1767),\n",
       " ('is', 1722),\n",
       " ('--', 1713),\n",
       " ('with', 1659),\n",
       " ('he', 1658),\n",
       " ('was', 1639),\n",
       " ('as', 1620)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "def answer_three():\n",
    "    order = sorted(dist.items(), key=lambda x: x[1],reverse = True)\n",
    "    return order[:20]\n",
    "\n",
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "\n",
    "What tokens have a length of greater than 5 and frequency of more than 150?\n",
    "\n",
    "*This function should return a sorted list of the tokens that match the above constraints. To sort your list, use `sorted()`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Captain',\n",
       " 'Pequod',\n",
       " 'Queequeg',\n",
       " 'Starbuck',\n",
       " 'almost',\n",
       " 'before',\n",
       " 'himself',\n",
       " 'little',\n",
       " 'seemed',\n",
       " 'should',\n",
       " 'though',\n",
       " 'through',\n",
       " 'whales',\n",
       " 'without']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_four():\n",
    "    freqwords = [k for k,v in dist.items() if len(k)>5 and v>150]\n",
    "    freqwords = sorted(freqwords)\n",
    "    return freqwords\n",
    "\n",
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5\n",
    "\n",
    "Find the longest word in text1 and that word's length.\n",
    "\n",
    "*This function should return a tuple `(longest_word, length)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"twelve-o'clock-at-night\", 23)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_five():\n",
    "    length = 0\n",
    "    longest_word = ''\n",
    "    for k,v in dist.items():\n",
    "        if len(k)>length:\n",
    "            length = len(k)\n",
    "            longest_word = k\n",
    "    return (longest_word,length)\n",
    "\n",
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6\n",
    "\n",
    "What unique words have a frequency of more than 2000? What is their frequency?\n",
    "\n",
    "\"Hint:  you may want to use `isalpha()` to check if the token is a word and not punctuation.\"\n",
    "\n",
    "*This function should return a list of tuples of the form `(frequency, word)` sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(13715, 'the'),\n",
       " (6513, 'of'),\n",
       " (6010, 'and'),\n",
       " (4545, 'a'),\n",
       " (4515, 'to'),\n",
       " (3908, 'in'),\n",
       " (2978, 'that'),\n",
       " (2459, 'his'),\n",
       " (2196, 'it'),\n",
       " (2097, 'I')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_six():\n",
    "    freq2000 = []\n",
    "    for k,v in dist.items():\n",
    "        if v>2000 and k.isalpha():\n",
    "            freq2000.append((v,k))\n",
    "    freq2000 = sorted(freq2000, key = lambda x: x[0],reverse = True)\n",
    "    return freq2000\n",
    "\n",
    "answer_six()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7\n",
    "\n",
    "What is the average number of tokens per sentence?\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25.881952902963864"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_seven():\n",
    "    sent = nltk.sent_tokenize(moby_raw)\n",
    "    per = len(text1)/len(sent)\n",
    "    return per\n",
    "\n",
    "answer_seven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 8\n",
    "\n",
    "What are the 5 most frequent parts of speech in this text? What is their frequency?\n",
    "\n",
    "*This function should return a list of tuples of the form `(part_of_speech, frequency)` sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('NN', 32730), ('IN', 28657), ('DT', 25867), (',', 19204), ('JJ', 17620)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eight():\n",
    "    dist2 = nltk.pos_tag(text1)\n",
    "    freq = {}\n",
    "    for i in dist2:\n",
    "            freq[i[1]] = freq.get(i[1],0)+1\n",
    "    freq = sorted(freq.items(), key = lambda x: x[1],reverse = True)    \n",
    "    return freq[:5]\n",
    "\n",
    "answer_eight()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 - Spelling Recommender\n",
    "\n",
    "For this part of the assignment you will create three different spelling recommenders, that each take a list of misspelled words and recommends a correctly spelled word for every word in the list.\n",
    "\n",
    "For every misspelled word, the recommender should find find the word in `correct_spellings` that has the shortest distance*, and starts with the same letter as the misspelled word, and return that word as a recommendation.\n",
    "\n",
    "*Each of the three different recommenders will use a different distance measure (outlined below).\n",
    "\n",
    "Each of the recommenders should provide recommendations for the three default words provided: `['cormulent', 'incendenece', 'validrate']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import words\n",
    "\n",
    "correct_spellings = words.words()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the trigrams of the two words.**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel/__main__.py:9: DeprecationWarning: generator 'ngrams' raised StopIteration\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['corpulent', 'indecence', 'validate']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_nine(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    l= len(entries)\n",
    "    d = [1]* l\n",
    "    recom = [w[0] for w in entries]\n",
    "    for i in correct_spellings:\n",
    "        for j in range(l):\n",
    "            if i.startswith(entries[j][0]):\n",
    "                a= set(nltk.ngrams(entries[j], n=3))\n",
    "                b= set(nltk.ngrams(i, n=3))\n",
    "                if nltk.jaccard_distance(a,b)< d[j]:\n",
    "                    d[j] = nltk.jaccard_distance(a,b)\n",
    "                    recom[j] = i\n",
    "    return recom\n",
    "    \n",
    "answer_nine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the 4-grams of the two words.**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel/__main__.py:9: DeprecationWarning: generator 'ngrams' raised StopIteration\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['cormus', 'incendiary', 'valid']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_ten(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    l= len(entries)\n",
    "    d = [1]* l\n",
    "    recom = [w[0] for w in entries]\n",
    "    for i in correct_spellings:\n",
    "        for j in range(l):\n",
    "            if i.startswith(entries[j][0]):\n",
    "                a= set(nltk.ngrams(entries[j], n=4))\n",
    "                b= set(nltk.ngrams(i, n=4))\n",
    "                if nltk.jaccard_distance(a,b)< d[j]:\n",
    "                    d[j] = nltk.jaccard_distance(a,b)\n",
    "                    recom[j] = i    \n",
    "    \n",
    "    return recom\n",
    "    \n",
    "answer_ten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 11\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Edit distance on the two words with transpositions.](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['corpulent', 'intendence', 'validate']"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eleven(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    l= len(entries)\n",
    "    recom = [w[0] for w in entries]\n",
    "    d = [nltk.edit_distance(recom[i],entries[i],transpositions = True) for i in range(l)]\n",
    "    for i in correct_spellings:\n",
    "        w=i.lower()\n",
    "        for j in range(l):\n",
    "            if w.startswith(entries[j][0]):\n",
    "                if nltk.edit_distance(w,entries[j],transpositions = True)< d[j]:\n",
    "                    d[j] = nltk.edit_distance(w,entries[j],transpositions = True)\n",
    "                    recom[j] = w          \n",
    "    return recom   \n",
    "answer_eleven()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-text-mining",
   "graded_item_id": "r35En",
   "launcher_item_id": "tCVfW",
   "part_id": "NTVgL"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}