Practical_2_chatbots_with_Regex.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00000-ffd9d2ec-fa74-4b58-ac2d-33cb822b34f8",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# Let's program a chatbot! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00001-b29473a6-a1df-4217-87b4-69d33e4f81e2",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "## What is a chatbot?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00002-71a05dfa-9e64-4ed3-93da-bfa7ea78a064",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "A **chatbot** is a software application:\n",
    "- used to conduct an on-line chat conversation via text or text-to-speech\n",
    "- supposed to replace a live human agent\n",
    "- designed to convincingly simulate the way a human would behave as a conversational partner\n",
    "- require continuous tuning and testing\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00003-f37b798b-099b-4233-87b9-ecf122284dde",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "** Applications**:\n",
    "- customer service\n",
    "- request routing\n",
    "- information gathering\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00005-4fe8d054-43d5-441a-a89e-1204a8a1ad18",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "![Picture title](img/image-20200917-165530.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00004-2c3dcace-7d5d-4ae1-9c7b-7a6522e74262",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "**Types**:\n",
    "- using extensive word-classification processes, natural language processors, and sophisticated AI\n",
    "- simply scanning for general keywords and generating responses using common phrases obtained from an associated library or database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00006-470259d7-4639-45d1-a049-1656b0edf3d5",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "1- **HOMEWORK QUESTION** (do your own research and answer): What is the difference between a chatbot, a socialbot and a virtual assistant?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00007-6cceb120-8898-4168-8bae-a5b939ab151f",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "<p style=\"color:green;\"><b>Type your response here<b><p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00007-f5f51480-397d-402c-8462-70ba5f14177d",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "## Turing test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00008-d01274e4-7ea6-4217-b9e6-ba97e3fbcda9",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "The **Turing test**, originally called **the imitation game** by Alan Turing in 1950, is a test of a machine's ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00009-14ee3342-ca50-49d6-a13f-afee2bf98e85",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "**IF YOU FANCY**: \n",
    "- Watch a great movie about Alan Turing and Enigma code: https://www.imdb.com/title/tt2084970/\n",
    "- Food for thought: would an AI purposefully hide its superintelligence in fear of being destroyed? Maybe it is intentionally failing Turing test? (if you are interested in this topic, \"Philosophy of AI\" is the general field this question belongs to)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00010-dd846c27-6ebd-4845-9f57-86a6cf723ca5",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "**Please watch a video about Turing test**: https://www.youtube.com/watch?v=3wLqsRLvV-c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00000-2176b628-1fde-47d9-9c7b-b36245d7df85",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "** Please watch a short video about Eliza**: https://www.youtube.com/watch?v=RMK9AphfLco\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00012-14618572-6765-4094-9fcd-fd8d776af930",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "**IF YOU FANCY**: Chat with Eliza if you feel like: https://web.njit.edu/~ronkowit/eliza.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00001-c273c6ed-42bb-49ff-8ad6-9bdcf580df6c",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "![Picture title](img/image-20200917-152808.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00014-b5290ef9-6ca2-4278-ab03-b3344160f3be",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# Edit distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00015-07038c58-6ceb-4453-9cbb-7849033eb063",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "**Edit distance** is a way of quantifying how dissimilar two strings (e.g. words) are to one another by counting the minimum number of operations required to transform one string into the other. **Levenshtein distance** is the most common metric and often used interchangeably with edit distance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00017-779e3fa4-33ca-4028-9378-1bb552dd7771",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "![Picture title](img/image-20200917-174435.png)\n",
    "Vladimir Levenstein, a Russian scientist who developed the Levenstein Distance algorithm in 1965."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00017-fd77c55f-6505-422b-aaa6-523ef93d3532",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "![Picture title](img/image-20200917-174109.png)\n",
    "(Image source (and a good read): https://medium.com/@ethannam/understanding-the-levenshtein-distance-equation-for-beginners-c4285a5604f0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00019-42b66771-671b-4115-ac72-d02fd6084f9c",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "**How do we use edit distance?**\n",
    "\n",
    "- computational biology/bioinformatics (comparing DNA sequences)\n",
    "- correction of spelling mistakes or OCR errors\n",
    "- approximate string matching, where the objective is to find matches for short strings in many longer texts, in situations where a small number of differences is to be expected (e.g. encryption).\n",
    "- auto suggestions of words\n",
    "- information rertieval\n",
    "- machine translation\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00020-47403530-7006-4fca-aa07-8c5fb5752e8f",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "### Spell Check example\n",
    "The following code queries using  name of a director on the ** IMDB dataset** and shows the **titles of their movies**.\n",
    "If the user inserts a wrong spelling it tries to match it to the directors it already knows using the Levenshtein Distance implemented in NLTK:\n",
    "\n",
    "https://python.gotrained.com/nltk-edit-distance-jaccard-distance/\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00021-9f479b2c-5883-4f76-85b3-af9221967c5d",
    "deepnote_cell_type": "code",
    "execution_millis": 870286,
    "execution_start": 1603186028414,
    "output_cleared": false,
    "source_hash": "8d1bdc9c",
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install nltk\n",
    "import pandas as pd #import Pandas library for data manipulation and analysis\n",
    "import nltk\n",
    "\n",
    "movies = pd.read_csv(\"IMDb movies.csv\")#read the file containing movies data\n",
    "\n",
    "#find directors with >10 movies , (to reduce search time)\n",
    "#create an array of unique values from the 'director' column\n",
    "directors = movies[movies.groupby('director')['director'].transform('size') > 10]['director'].unique()\n",
    "\n",
    "quit =False\n",
    "while(not quit):\n",
    "    \n",
    "    input_name = input(\"Enter a director's full name(example: David Lynch):Press q to quit)\")\n",
    "    quit = (input_name==\"q\")\n",
    "    if(not quit):\n",
    "        if(input_name in directors):\n",
    "            #find movies of the specified director\n",
    "            movie_names = movies.loc[movies['director'] == input_name]['original_title']\n",
    "            print(\"Here are the name of movies I know directed by \" +input_name +\":\")\n",
    "            display(movie_names.reset_index(drop=True))\n",
    "\n",
    "        #handle the misspelled director's name:\n",
    "        else:\n",
    "          \n",
    "\n",
    "            distances = [nltk.edit_distance(input_name,director) for director in directors] #computing edit distance of the input name if not found in the list with all directors\n",
    "            \n",
    "\n",
    "            guessed = directors[distances.index(min(distances))] #getting minimum distance director's name with the input\n",
    "            answer = input(\"Did you mean \"+ guessed+\"?(y/n))\") # asking feedback from the user if we found the correct name\n",
    "            if(answer.lower() in [\"yes\",\"y\"]):\n",
    "                movie_names = movies.loc[movies['director'] == guessed]['original_title'] # looking for movies from the director if the guess was correct\n",
    "                print(\"Here are the name of movies I know directed by \" +guessed +\":\")\n",
    "                display(movie_names.reset_index(drop=True))\n",
    "            elif(answer ==\"q\"):# checking if user already wants to exit program\n",
    "                quit = True\n",
    "                continue;\n",
    "            else:\n",
    "                print(\"I might not know this director! sorry\") # Giving up on first wrong guess!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00022-dc3f294e-a429-44de-8bba-bdc153237b81",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "2- **Observe and Reflect**: What happens if we only look for the last name of a director? For example : _Linch_ instead of _Lynch_\n",
    "\n",
    "- how do you suggest we fix problem?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00023-97fe7e54-eaee-4de3-8271-1bda2d39b195",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "<p style=\"color:green;\"><b>Type your response here<b><p>\n",
    "\n",
    "\n",
    "We can look for names and last names separately and match them to our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00024-b5bafd50-daca-4e19-9acf-83ee2df83636",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "3- **Observe and Reflect**: \n",
    "Read about Jaccard distance [here](https://python.gotrained.com/nltk-edit-distance-jaccard-distance/)\n",
    "\n",
    "Can you think of an example where **Jaccard distance** might give you the correct guess but **Levenshtein distance** doesn't? \n",
    "Explain why.\n",
    "\n",
    "(Go ahead and try it in code)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00025-b992197a-356b-4ff3-a727-714d875727b5",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "<p style=\"color:green;\"><b>Type your response here<b><p>\n",
    "\n",
    "Jaccard distance does not consider the position of the characters in computing the distance, so if the characters all exist but they are in the wrong order Jaccard distance find the closest match , however since the Levenshtein distance only considers insert , delete and substitute as actions , if the characters are all in wrong positions the edit distance would be very high and not close to the match.\n",
    "\n",
    "\n",
    "consider this pattern for example: Chyn DaLdiv\n",
    "\n",
    "``` distances = [nltk.jaccard_distance(set(input_name),set(director)) for director in directors] #computing edit distance of the input name if not found in the list with all directors ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00026-89a78037-d59e-4a91-8aaf-08b6a5857988",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00021-1432f4bc-46bb-4ed4-89d6-407b27c2aa68",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "## Regex Warm up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00022-8dc8fe5c-8796-49fa-aca2-e264fff3ead1",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "There are platforms to test regex such as:\n",
    "https://regexr.com/\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00023-67ad30fd-9d5a-4cd2-a63a-572f03e4d208",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "### regex.findall() \n",
    "**regex.findall** function finds all patterns matching the regex.\n",
    "\n",
    "\n",
    "See an example of tokenization using regex in the following code cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00025-5e24d540-e19c-4f85-93d4-13126aa08aec",
    "deepnote_cell_type": "code",
    "output_cleared": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import regex\n",
    "text = \"Hello, I am studying computer science and this is my NLP course material.\"\n",
    "regex.findall(\"\\w+\",text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00026-110f6b29-60de-4de6-a96b-ba5f16d2db78",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "### regex.sub\n",
    "Function **regex.sub** replaces a text with the patterns matched to the regex.\n",
    "\n",
    "See an example in the following code cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00023-1463459e-55e9-40a5-84d4-8fc5881c72df",
    "deepnote_cell_type": "code",
    "output_cleared": false,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import regex\n",
    "text = \"Hello, I am studying computer science and this is my NLP course material. Yay!! \"\n",
    "text = regex.sub(r\"[\\W]\",\" \",text) #replace non alphabetical and numbers with space\n",
    "text =regex.sub(r\"\\s+\",\" \",text)#replace every sequences of spaces with only one space\n",
    "\n",
    "print(text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00028-d13bc3e6-03a0-4e42-8c34-0ade3024fee3",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "### regex.match \n",
    "Function **regex.match** puts parts of the matched regex which are distinguished by paranthesis in different groups.\n",
    "\n",
    "See an example identifying a specific time pattern in the following code cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00028-e3e11b8f-a200-490d-9b47-384a748a5f44",
    "deepnote_cell_type": "code",
    "output_cleared": false,
    "tags": []
   },
   "outputs": [],
   "source": [
    "time = \"12:05 PM Tuesday, September 01, 2020 (GMT+2)\"\n",
    "\n",
    "groups = regex.match(\"(\\d{2}):(\\d{2})\",time)# \\d matches digit and using {} you can choose the number of repetitions for instance \\w{2-5} mins minimum 2 alphabetical characters and maximum 5 alphabetical characters\n",
    "matched = groups[0]\n",
    "hour = groups[1]\n",
    "minutes = groups[2]\n",
    "print(hour,\"h\",minutes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00029-ef964681-487f-4ea5-8419-f114f70ba4a8",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "4- **Observe and Reflect**  Please give an example of a pattern that can be considered by the regex correct in the example above (when extracting time) even though they are in fact not correct?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00029-8da8fe65-0f2b-45b8-afd9-e188c6eda901",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "<p style=\"color:green;\"><b>Type your response here<b><p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00030-27be8871-5fdb-460d-9fb9-177670518cba",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "5- **CODEIT** Edit the regex in the above code cell to  fix the problem you found with the pattern?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00038-cf035026-24ad-441b-8564-04b8f610f300",
    "deepnote_cell_type": "code",
    "output_cleared": false,
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "time = \"12:05 PM Tuesday, September 01, 2020 (GMT+2)\"\n",
    "\n",
    "groups = regex.match(\"([0-1][0-9]|[2][0-3]):([0-5][0-9])\",time)\n",
    "\n",
    "matched = groups[0]\n",
    "hour = groups[1]\n",
    "minutes = groups[2]\n",
    "print(hour,\"h\",minutes)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00031-86c5b9b5-af47-4b16-bbf9-865811b7d7db",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00029-fa883300-ccff-4c38-8185-a7f45b402d11",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "6- **CODE IT** In the following text from \"Alice in wonderland\" find where in text someone is quoted and their quote (as best as you can) by filling the regex in the match function.\n",
    "\n",
    " Note: make sure you use correct quotation characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00030-61e0910c-3b0a-4126-b288-1b7a6c0b5505",
    "deepnote_cell_type": "code",
    "output_cleared": false,
    "tags": []
   },
   "outputs": [],
   "source": [
    "text ='''“Ugh!” said the Lory, with a shiver.\n",
    "\n",
    "“I beg your pardon!” said the Mouse, frowning, but very politely: “Did\n",
    "you speak?”\n",
    "\n",
    "“Not I!” said the Lory hastily.\n",
    "\n",
    "“I thought you did,” said the Mouse. “—I proceed. ‘Edwin and Morcar,\n",
    "the earls of Mercia and Northumbria, declared for him: and even\n",
    "Stigand, the patriotic archbishop of Canterbury, found it advisable—’”\n",
    "\n",
    "“Found _what_?” said the Duck.\n",
    "\n",
    "“Found _it_,” the Mouse replied rather crossly: “of course you know\n",
    "what ‘it’ means.”\n",
    "\n",
    "“I know what ‘it’ means well enough, when _I_ find a thing,” said the\n",
    "Duck: “it’s generally a frog or a worm. The question is, what did the\n",
    "archbishop find?”\n",
    "\n",
    "The Mouse did not notice this question, but hurriedly went on, “‘—found\n",
    "it advisable to go with Edgar Atheling to meet William and offer him\n",
    "the crown. William’s conduct at first was moderate. But the insolence\n",
    "of his Normans—’ How are you getting on now, my dear?” it continued,\n",
    "turning to Alice as it spoke.\n",
    "\n",
    "“As wet as ever,” said Alice in a melancholy tone: “it doesn’t seem to\n",
    "dry me at all.”\n",
    "\n",
    "“In that case,” said the Dodo solemnly, rising to its feet, “I move\n",
    "that the meeting adjourn, for the immediate adoption of more energetic\n",
    "remedies—”\n",
    "\n",
    "“Speak English!” said the Eaglet. “I don’t know the meaning of half\n",
    "those long words, and, what’s more, I don’t believe you do either!” And\n",
    "the Eaglet bent down its head to hide a smile: some of the other birds\n",
    "tittered audibly.\n",
    "\n",
    "“What I was going to say,” said the Dodo in an offended tone, “was,\n",
    "that the best thing to get us dry would be a Caucus-race.”\n",
    "\n",
    "“What _is_ a Caucus-race?” said Alice; not that she wanted much to\n",
    "know, but the Dodo had paused as if it thought that _somebody_ ought to\n",
    "speak, and no one else seemed inclined to say anything.\n",
    "\n",
    "“Why,” said the Dodo, “the best way to explain it is to do it.” (And,\n",
    "as you might like to try the thing yourself, some winter day, I will\n",
    "tell you how the Dodo managed it.)'''\n",
    "\n",
    "\n",
    "def findSpeakerAndQuote(sentence):\n",
    "    speaker = \"\"\n",
    "    quote = \"\"\n",
    "    try:\n",
    "        print(sentence)\n",
    "        #CODEIT: insert regex in match function below to find the speaker and the quotes in the above text as best as you can\n",
    "        grouped = regex.match(r\"(.*?) said ((the\\s)?(\\w+))\",sentence)\n",
    "        speaker = grouped[2]\n",
    "        quote = grouped[1]\n",
    "        print(speaker+\" : \"+quote)\n",
    "        print(\"_\"*20)\n",
    "        \n",
    "    except:\n",
    "        pass\n",
    "    \n",
    "        \n",
    "import regex\n",
    "\n",
    "\n",
    "paragraphs = text.split(\"\\n\\n\")\n",
    "speaker_quotes=[findSpeakerAndQuote(paragraph) for paragraph in paragraphs]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00023-e829a3ac-cdb4-4932-b5b1-3925b0fbd1ab",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "7- **CODE IT**  The following code is supposed to extract hashtags from a tweet. Find all hashtags in the tweet using \"findHashtags\" regex and return them as a list.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00024-58f168de-f196-4146-b927-67340bb6565a",
    "deepnote_cell_type": "code",
    "execution_millis": 550,
    "execution_start": 1603186898720,
    "output_cleared": false,
    "source_hash": "c4ca4204",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import regex\n",
    "import pandas as pd #importing pandas to work with dataframes\n",
    "\n",
    "def findHashtags(tweet): \n",
    "    #CODEIT: Insert Regex pattern to return the list of hashtags in tweet\n",
    "    \n",
    "    hashtags = regex.findall(\"#\\w+\",tweet)\n",
    "    \n",
    "    return hashtags\n",
    "    \n",
    "tweets = pd.read_csv(\"Tweets.csv\")['Tweet'].tolist()\n",
    "text = tweets[0]\n",
    "print(text)\n",
    "hashtags=findHashtags(text)\n",
    "hashtags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00025-4321deb7-7c6f-439a-bd83-90139868c3eb",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "If if complete the exercise above correctly, the following code will create 2 wordclouds of the hashtags used by Republicans and Democrates separately. Run the code below and wait."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00045-f3572897-85c1-44be-8250-e55d161797a8",
    "deepnote_cell_type": "code",
    "execution_millis": 3155,
    "execution_start": 1603188520890,
    "output_cleared": false,
    "source_hash": "4b787ab3",
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install wordcloud==1.8.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "allow_embed": false,
    "cell_id": "00025-f29d92cb-4626-4e1b-832d-d24f41807163",
    "deepnote_cell_type": "code",
    "execution_millis": 7230,
    "execution_start": 1603188530364,
    "output_cleared": false,
    "source_hash": "e5944e4b",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd #importing pandas to work with dataframes\n",
    "\n",
    "\n",
    "tweets = pd.read_csv(\"Tweets.csv\")\n",
    "democrats_tweets = tweets.loc[tweets['Party'] == \"Democrat\"]['Tweet'].tolist()\n",
    "republicans_tweets = tweets.loc[tweets['Party'] == \"Republican\"]['Tweet'].tolist()\n",
    "\n",
    "import regex\n",
    "\n",
    "all_democrat_hashtags=[]\n",
    "for tweet in democrats_tweets:\n",
    "    all_democrat_hashtags.extend(findHashtags(tweet))\n",
    "    \n",
    "\n",
    "all_republican_hashtags=[]\n",
    "for tweet in republicans_tweets:\n",
    "    all_republican_hashtags.extend(findHashtags(tweet))\n",
    "    \n",
    "\n",
    "\n",
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    " \n",
    "!pip install wordcloud\n",
    "wordcloud_rep = WordCloud().generate(\" \".join(all_republican_hashtags))\n",
    "wordcloud_dem = WordCloud().generate(\" \".join(all_democrat_hashtags))\n",
    " \n",
    "# Display the generated image:\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(2,1,1)\n",
    "ax.imshow(wordcloud_rep, interpolation='bilinear')\n",
    "ax = fig.add_subplot(2,1,2)\n",
    "ax.imshow(wordcloud_dem, interpolation='bilinear')\n",
    "plt.axis(\"off\")\n",
    "plt.margins(x=5, y=5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00035-f143c8c3-9714-4a1b-89ef-262aaa8fc4ed",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "## Create your own chatbot with Python\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00036-ab1cd499-2f41-486e-a055-835162c2bc6d",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "The following code creates a chatbot using nltk library.\n",
    "\n",
    "Chat with Chatty a bit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00002-c9ddea7c-8dee-4362-be2f-d438c8d70e24",
    "deepnote_cell_type": "code",
    "execution_millis": 52389,
    "execution_start": 1603187073554,
    "output_cleared": false,
    "source_hash": "a3ab0d9f",
    "tags": []
   },
   "outputs": [],
   "source": [
    "from nltk.chat.util import Chat,reflections\n",
    "pairs = [[r\"my name is (.*)\",[\"Hello %1, How are you today ?\",] ],\\\n",
    "                                                    [r\"what is your name ?\", [\"My name is Chatty and I'm a chatbot ?\",]],\\\n",
    "                                                    [r\"how are you ?\",[\"I'm doing good\\nHow about You ?\",]],\\\n",
    "                                                    [r\"sorry (.*)\",  [\"It's alright\",\"It's OK, never mind\",]],\\\n",
    "                                                    [r\"i'm not( .*)? doing good\",[\"I'm sorry\",]],\\\n",
    "\n",
    "                                                    [r\"i'm (.*) doing good\",[\"Nice to hear that\",\"Alright :)\",]],\\\n",
    "                                                    [r'(.*) (hungry|sleepy)', [\"Are you saying %1 %2 ?\"]],\\\n",
    "                                                    [r\"hi|hey|hello\",[\"Hello\", \"Hey there\",]],\\\n",
    "                                                    [r\"(.*) age?\",[\"I'm a computer program dude\\nSeriously you are asking me this?\",]],\\\n",
    "                                                    [r\"what (.*) want ?\",[\"Make me an offer I can't refuse\",]],\\\n",
    "                                                    [r\"how is weather in (.*)?\",[\"Weather in %1 is awesome like always\",\"Too hot man here in %1\",\"Too cold man here in %1\",\"Never even heard about %1\"]],\\\n",
    "                                                    [r\"i work in (.*)?\", [\"%1 is an Amazing company, I have heard about it. But they are in huge loss these days.\",]],\\\n",
    "                                                    [r\"(.*)raining in (.*)\", [\"No rain since last week here in %2\",\"Damn its raining too much here in %2\"]],\\\n",
    "                                                    [r\"how (.*) health(.*)\",[\"I'm a computer program, so I'm always healthy \",]],\\\n",
    "                                                    [r\"quit\",[\"BBye take care. See you soon :) \",\"It was nice talking to you. See you soon :)\"]]]\n",
    "reflections = {  \"i am\"       : \"you are\",  \"i was\"      : \"you were\",  \"i\"          : \"you\",  \"i'm\"        : \"you are\",  \"i'd\"        : \"you would\",  \"i've\"       : \"you have\",  \"i'll\"       : \"you will\",  \"my\"         : \"your\",  \"you are\"    : \"I am\",  \"you were\"   : \"I was\",  \"you've\"     : \"I have\",  \"you'll\"     : \"I will\",  \"your\"       : \"my\",  \"yours\"      : \"mine\",  \"you\"        : \"me\",  \"me\"         : \"you\"}\n",
    "def chatty():\n",
    "    print(\"Hi, I'm Chatty and I chat alot ;)\\nPlease type lowercase English language to start a conversation. Type quit to leave \") #default message at the start \n",
    "    chat = Chat(pairs, reflections)\n",
    "    chat.converse()\n",
    "\n",
    "chatty()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00038-d34206fe-0e16-47c4-9cc7-a9923fd5b69d",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "There is a chance if you tell chatty :\"i'm not doing good\" , it will response with: \"Nice to hear that\" . Try it!\n",
    "\n",
    "8- **CODE IT** add a pair to the pairs list of chatty in the above code to fix that with feeling sorry for someone who is not feeling good."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00042-67d22b38-8b8b-47aa-89d7-9330f3e78f5e",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00038-8191dd0b-bb55-4d02-b048-bfaa68bc4c58",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "9- **Homework Exercise:**\n",
    "Choose one of the following chatbot scenarios and create it using the nltk library in python.\n",
    "\n",
    "- Movie recommendation chatbot : Your chatbot should know at least 100 movies from the IMDB Movie dataset in different genres.\n",
    "> Sample Coversation for Movie chatbot:\n",
    "``` User: Hello chatty\n",
    "    Chatty: Hi there\n",
    "    User: How are you?\n",
    "    Chatty: I am doing fine, How are you?\n",
    "    User: Fine, Fine.\n",
    "    Chatty: Glad to hear.\n",
    "    User: Can you recommend me a movie to watch? (I want to watch a film.)(Can you recommend a movie?)(I am looking for a good movie)\n",
    "    Chatty: of course, What genre do you fancy? (Do you have a specific genre in mind?) \n",
    "    User: Comdey?\n",
    "    Chatty: How about Sweet and Lowdown? It's a comedy movie directed by Woody Allen in 1999\n",
    "    User: I have seen this one, do you no any other comedy movies?\n",
    "    Chatty: How about The Angels' Share? It's a comedy movie directed by Ken Loach in 2012\n",
    "    User: Thanks Chatty\n",
    "    Chatty: You are very welcome.\n",
    "    User: Bye\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "- Travel Agency chatbot :Use a list of cities with some explanations about the city extracted from a list.\n",
    "> Sample Coversation for Travel Agency chatbot:\n",
    "```User: Hello chatty\n",
    "    Chatty: Hi there\n",
    "    User: How are you?\n",
    "    Chatty: I am doing fine, How are you?\n",
    "    User: Fine, Fine.\n",
    "    Chatty: Glad to hear.\n",
    "    User: I want to go on a trip. (I am thinking of traveling.)\n",
    "    Chatty: How nice, where are you going?\n",
    "    User: I don't know yet. Where do you recommend? (Where do you suggest?)\n",
    "    Chatty: How about Paris? You can spend a lot of time in museums.\n",
    "    User: I have been there before. Anywhere else? (where else?) (where else do you know about?)\n",
    "    Chatty: How about Montreal? it's a beautiful city in canda with night life.\n",
    "    User: Sounds good/fine/super.\n",
    "    Chatty: I'm happy you like my recommendation.\n",
    "    User: I want to leave on Wednesday 12th of August.(on Sunday, Next weekend , 12/08/2020, ...(anyother time format that comes to your mind...))\n",
    "    Chatty: Ok I'll contact the agency for a ticket on 12th of August and get back to you(fake promise!).\n",
    "    User: Bye\n",
    "```\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "deepnote_execution_queue": [],
  "deepnote_notebook_id": "6530a535-5a89-468b-9e42-db4e49dd2d3b",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}