Descriptive-statistics-distribution-PS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Copyright 2021 Andrew M. Olney and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Descriptive statistics distribution-based metrics: Problem solving\n",
    "\n",
    "In this notebook, we will look at differences between Trump and AOC tweets.\n",
    "The data for this was obtained [here](https://github.com/dbamman/anlp19/tree/master/data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and preprocess the data\n",
    "\n",
    "NLTK has a special corpus reader for tweets, but the reader assumes that each tweet is on its own line.\n",
    "In contrast, our data is all on a single line (as a list) in a file for Trump and a file for AOC.\n",
    "We will use the `json` library to read these files.\n",
    "We will then use a comprehension to get the data we want back out.\n",
    "\n",
    "Import the `json` library and `nltk`.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk as nltk\n",
    "import json as json\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable><variable id=\"SMiSZAoKudUu:H7fDUDl\">json</variable></variables><block type=\"importAs\" id=\"_Xys=P-k-4H|=dpb09rV\" x=\"53\" y=\"67\"><field name=\"libraryName\">nltk</field><field name=\"libraryAlias\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><next><block type=\"importAs\" id=\":;vvV%],n8`)D0u|x!!S\"><field name=\"libraryName\">json</field><field name=\"libraryAlias\" id=\"SMiSZAoKudUu:H7fDUDl\">json</field></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to load the contents of `trump_tweets.json` and store in `trumpData`.\n",
    "\n",
    "You've already seen how to read a text file into a variable, but this is a little different.\n",
    "\n",
    "To interpret JSON data, you will need to use with `json` do `load` followed by `read file`.\n",
    "\n",
    "Once you've got it, do the same for `aoc_tweets.json` and store in `aocData`. \n",
    "You can check your work if you need to by looking at the first tweet in either list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "trumpData = json.load(open('datasets/trump_tweets.json',encoding='utf-8'))\n",
    "aocData = json.load(open('datasets/aoc_tweets.json',encoding='utf-8'))\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"k?0pC,e]Q9`cf5_~e|s0\">trumpData</variable><variable id=\"SMiSZAoKudUu:H7fDUDl\">json</variable><variable id=\"H;AOj:jE/[^%4O+.5/:|\">aocData</variable></variables><block type=\"variables_set\" id=\"e|V8U`kBsrjh)Hwu9JA?\" x=\"36\" y=\"86\"><field name=\"VAR\" id=\"k?0pC,e]Q9`cf5_~e|s0\">trumpData</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"[i;]zJF|*wtQ.$vj@eB!\"><field name=\"VAR\" id=\"SMiSZAoKudUu:H7fDUDl\">json</field><field name=\"MEMBER\">load</field><data>json:load</data><value name=\"INPUT\"><block type=\"readFile\" id=\"*?*?1fXBVDw)~YADk9IB\"><field name=\"FILENAME\">datasets/trump_tweets.json</field></block></value></block></value><next><block type=\"variables_set\" id=\"D%4_3_ew*}ic%gIOK)U[\"><field name=\"VAR\" id=\"H;AOj:jE/[^%4O+.5/:|\">aocData</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"Z)8/N:/I$dx8op^c{LqE\"><field name=\"VAR\" id=\"SMiSZAoKudUu:H7fDUDl\">json</field><field name=\"MEMBER\">load</field><data>json:load</data><value name=\"INPUT\"><block type=\"readFile\" id=\"alZw{szR)44{5uXw{hY`\"><field name=\"FILENAME\">datasets/aoc_tweets.json</field></block></value></block></value></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------\n",
    "**QUESTION:**\n",
    "\n",
    "What is the difference between reading a text file and opening a file for reading?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER: (click here to edit)**\n",
    "\n",
    "*Reading a text file gives you a string. Opening a file for reading gives you a reference to the file but doesn't do any reading.*\n",
    "\n",
    "----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have lists of raw tweets. \n",
    "As we've previously seen, we only care about the `text` field of these and not all the metadata.\n",
    "\n",
    "We can get just the text using a comprehension that takes `trumpData` and yields `i[\"text\"]` (assuming the comprehension looping variable is `i`).\n",
    "`i[\"text\"]` gives use the field we want because `json.load` has given us a list of JSON objects, and we can treat each object like a **dictionary**.\n",
    "\n",
    "Create this list comprehension and store the resulting list in `trumpTweetTexts`.\n",
    "When you've got it, do the same for `aocTweetTexts`.\n",
    "You can check your work if you need to by looking at the first tweet in either list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "aocTweetTexts = [i['text'] for i in aocData]\n",
    "trumpTweetTexts = [i['text'] for i in trumpData]\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"#j(I~[G2QP;5Y=$!)m-!\">aocTweetTexts</variable><variable id=\"Ndz1o_}c{0XFKqh|-M+-\">trumpTweetTexts</variable><variable id=\"(f;.!_O`Eek@)_YwJHY!\">i</variable><variable id=\"H;AOj:jE/[^%4O+.5/:|\">aocData</variable><variable id=\"k?0pC,e]Q9`cf5_~e|s0\">trumpData</variable></variables><block type=\"variables_set\" id=\"XUPb:)J-N7uyN2$p09L9\" x=\"16\" y=\"54\"><field name=\"VAR\" id=\"#j(I~[G2QP;5Y=$!)m-!\">aocTweetTexts</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"=4hBa^54q.e+:tIsA@/,\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"bQIEF.w3*-SAZwA{sv1f\"><field name=\"VAR\" id=\"(f;.!_O`Eek@)_YwJHY!\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"pW5}k24JvNW]8d8?tFBR\"><field name=\"VAR\" id=\"H;AOj:jE/[^%4O+.5/:|\">aocData</field></block></value><value name=\"YIELD\"><block type=\"indexer\" id=\"gMU*fAhE6*PQ9]RCE-kc\"><field name=\"VAR\" id=\"(f;.!_O`Eek@)_YwJHY!\">i</field><value name=\"INDEX\"><block type=\"text\" id=\"xgGX8_k!a5O)SBA][$5|\"><field name=\"TEXT\">text</field></block></value></block></value></block></value></block></value><next><block type=\"variables_set\" id=\",JhO6dN]XvuFt^Y03I)d\"><field name=\"VAR\" id=\"Ndz1o_}c{0XFKqh|-M+-\">trumpTweetTexts</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"*B+%(GY,}/8v.djFP(Ld\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"8q$Zk4_|^|3J^kWIi05a\"><field name=\"VAR\" id=\"(f;.!_O`Eek@)_YwJHY!\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"0Jt@{wAPhpcM:6{;3Sss\"><field name=\"VAR\" id=\"k?0pC,e]Q9`cf5_~e|s0\">trumpData</field></block></value><value name=\"YIELD\"><block type=\"indexer\" id=\"!jsCf|TUFKW7*JY$|N_1\"><field name=\"VAR\" id=\"(f;.!_O`Eek@)_YwJHY!\">i</field><value name=\"INDEX\"><block type=\"text\" id=\"DXR^!NrMc#bogEQ=fdeL\"><field name=\"TEXT\">text</field></block></value></block></value></block></value></block></value></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have the text, but we need word tokens to do our distribution-based metrics.\n",
    "\n",
    "So create a `TweetTokenizer` using nltk and store it in variable `tweetTokenizer`.\n",
    "To make everything lower case, create it using `preserve_case=False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweetTokenizer = nltk.TweetTokenizer(preserve_case=False)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable></variables><block type=\"variables_set\" id=\"M6T0am_1LQ~%mS$Po-v^\" x=\"-4\" y=\"179\"><field name=\"VAR\" id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"A40ZnNV*DXCia`@$Xj`b\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">TweetTokenizer</field><data>nltk:TweetTokenizer</data><value name=\"INPUT\"><block type=\"dummyOutputCodeBlock\" id=\"?C}k8uVXGK#Ji?@qpQ_`\"><field name=\"CODE\">preserve_case=False</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to tokenize both `trumpTweetTexts` and `aocTweetTexts` using comprehensions and store the results in respective variables `trumpTokens` and `aocTokens`.\n",
    "\n",
    "However, if we tokenize each element of our list of tweets, we will end up with a list of lists of tokens (each tweet will be a list of tokens, and we'll have one big list containing all the tweets).\n",
    "\n",
    "To use NLTK's distribution-based metrics, we just need one giant list of words.\n",
    "We can use NLTK's `flatten` operation to do this: `flatten` collapses nested lists into a single list.\n",
    "So after you get `trumpTokens` above, set it to with `nltk` do `flatten` `trumpTokens` to get a flat list.\n",
    "Do the same for `aocTokens`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "trumpTokens = [(tweetTokenizer.tokenize(i)) for i in trumpTweetTexts]\n",
    "trumpTokens = nltk.flatten(trumpTokens)\n",
    "aocTokens = [(tweetTokenizer.tokenize(i)) for i in aocTweetTexts]\n",
    "aocTokens = nltk.flatten(aocTokens)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</variable><variable id=\"(f;.!_O`Eek@)_YwJHY!\">i</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable><variable id=\"69-pKS_JOb@!NE-8XbA]\">aocTokens</variable><variable id=\"Ndz1o_}c{0XFKqh|-M+-\">trumpTweetTexts</variable><variable id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</variable><variable id=\"#j(I~[G2QP;5Y=$!)m-!\">aocTweetTexts</variable></variables><block type=\"variables_set\" id=\"3l|EfbruO{4mf5lVXXg`\" x=\"55\" y=\"531\"><field name=\"VAR\" id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"+r6y%@fc/`{1j;de3U@N\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"4*|:q[+BkLb2[2tH),%,\"><field name=\"VAR\" id=\"(f;.!_O`Eek@)_YwJHY!\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"8[TQc:/%s#ZaLZCKM(X_\"><field name=\"VAR\" id=\"Ndz1o_}c{0XFKqh|-M+-\">trumpTweetTexts</field></block></value><value name=\"YIELD\"><block type=\"varDoMethod\" id=\"v2PS!e`spD#%0nQ{V[1.\"><field name=\"VAR\" id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</field><field name=\"MEMBER\">tokenize</field><data>trumpTokens:</data><value name=\"INPUT\"><block type=\"variables_get\" id=\";jG,GnEfZ/-fY5,rl*t_\"><field name=\"VAR\" id=\"(f;.!_O`Eek@)_YwJHY!\">i</field></block></value></block></value></block></value></block></value><next><block type=\"variables_set\" id=\"TjE-tuPb!BXDuJ!OU/@d\"><field name=\"VAR\" id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"UY:b7+7Hq!2#UafnJUTv\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">flatten</field><data>nltk:flatten</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"`a~rgC#4!`e0``CR-Wzc\"><field name=\"VAR\" id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</field></block></value></block></value><next><block type=\"variables_set\" id=\".mw)0OloOXt?`bG=Y=}Y\"><field name=\"VAR\" id=\"69-pKS_JOb@!NE-8XbA]\">aocTokens</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"zd0FDB`{RVXRdR3`h]/c\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"j^x}jL-^He*_x6^CR1$5\"><field name=\"VAR\" id=\"(f;.!_O`Eek@)_YwJHY!\">i</field><value name=\"LIST\"><block type=\"variables_get\" id=\"%`-$B_B{_X57g4TF[S+K\"><field name=\"VAR\" id=\"#j(I~[G2QP;5Y=$!)m-!\">aocTweetTexts</field></block></value><value name=\"YIELD\"><block type=\"varDoMethod\" id=\"/;3oK}H;QC8UPTKDFYsi\"><field name=\"VAR\" id=\"=tk`p3@7pVEk8zIAf7@;\">tweetTokenizer</field><field name=\"MEMBER\">tokenize</field><data>trumpTokens:</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"39}X2$;d@r6TZk(4l|-~\"><field name=\"VAR\" id=\"(f;.!_O`Eek@)_YwJHY!\">i</field></block></value></block></value></block></value></block></value><next><block type=\"variables_set\" id=\"-=/t)rm|O6}Q6rBN2ZS@\"><field name=\"VAR\" id=\"69-pKS_JOb@!NE-8XbA]\">aocTokens</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"g2N-`Ogo16u^O1qZLdpo\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">flatten</field><data>nltk:flatten</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"68tf@Db_oQ~yMU/*oQ|o\"><field name=\"VAR\" id=\"69-pKS_JOb@!NE-8XbA]\">aocTokens</field></block></value></block></value></block></next></block></next></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distribution-based metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lexical diversity\n",
    "\n",
    "Let's compare type/token ratio for Trump and AOC.\n",
    "\n",
    "Calculate for Trump:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.058665549274307574"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(trumpTokens)) / len(trumpTokens)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</variable></variables><block type=\"math_arithmetic\" id=\",a5|i%aYs{5_dwD`]j2{\" x=\"-12\" y=\"60\"><field name=\"OP\">DIVIDE</field><value name=\"A\"><shadow type=\"math_number\" id=\"Sf$2}-h2AT#|4/5lQUkT\"><field name=\"NUM\">1</field></shadow><block type=\"lists_length\" id=\"QaTajcn{EsW9G],%Ri3!\"><value name=\"VALUE\"><block type=\"setBlock\" id=\"|!f*)[[6.{POaT4A#ime\"><value name=\"x\"><block type=\"variables_get\" id=\"ys5AOym%ofGO$:MbK#tH\"><field name=\"VAR\" id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</field></block></value></block></value></block></value><value name=\"B\"><shadow type=\"math_number\" id=\"Tmf?bHPN}[P*T3b6(|O{\"><field name=\"NUM\">1</field></shadow><block type=\"lists_length\" id=\"^2]p$ss.Y@kU=B#X1pwI\"><value name=\"VALUE\"><block type=\"variables_get\" id=\"-wk,o[;i?x}w~=Y*yB,!\"><field name=\"VAR\" id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And AOC:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.14699276266533567"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(aocTokens)) / len(aocTokens)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"69-pKS_JOb@!NE-8XbA]\">aocTokens</variable></variables><block type=\"math_arithmetic\" id=\",a5|i%aYs{5_dwD`]j2{\" x=\"-12\" y=\"60\"><field name=\"OP\">DIVIDE</field><value name=\"A\"><shadow type=\"math_number\" id=\"Sf$2}-h2AT#|4/5lQUkT\"><field name=\"NUM\">1</field></shadow><block type=\"lists_length\" id=\"QaTajcn{EsW9G],%Ri3!\"><value name=\"VALUE\"><block type=\"setBlock\" id=\"|!f*)[[6.{POaT4A#ime\"><value name=\"x\"><block type=\"variables_get\" id=\"ys5AOym%ofGO$:MbK#tH\"><field name=\"VAR\" id=\"69-pKS_JOb@!NE-8XbA]\">aocTokens</field></block></value></block></value></block></value><value name=\"B\"><shadow type=\"math_number\" id=\"Tmf?bHPN}[P*T3b6(|O{\"><field name=\"NUM\">1</field></shadow><block type=\"lists_length\" id=\"^2]p$ss.Y@kU=B#X1pwI\"><value name=\"VALUE\"><block type=\"variables_get\" id=\"-wk,o[;i?x}w~=Y*yB,!\"><field name=\"VAR\" id=\"69-pKS_JOb@!NE-8XbA]\">aocTokens</field></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------\n",
    "**QUESTION:**\n",
    "\n",
    "What do these lexical diversity results mean?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER: (click here to edit)**\n",
    "\n",
    "*Maximum lexical diversity occurs when every word is different from every other word.\n",
    "When that happens, type/token ratio is 1. So higher TTR means more lexical diversity.\n",
    "AOC's lexical diversity is approximately three times Trump's.\n",
    "This could mean that she is covering more topics in her speech or is using more synonyms or both.*\n",
    "\n",
    "----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequency distributions\n",
    "\n",
    "Let's find some distinctive words for Trump and AOC.\n",
    "We previously saw that using the top 50 words wasn't that interesting because they were function words.\n",
    "Common uninteresting words are sometimes called **stop words**.\n",
    "NLTK has a stop word list, so we can remove them from our frequency distributions.\n",
    "\n",
    "In addition to stop words, we have another issue to resolve - there are many more Trump tweets than AOC tweets.\n",
    "Later on, we're going to make direct numerical comparisons, so let's go ahead and truncate the Trump data to have the same length as AOC.\n",
    "\n",
    "Import `stopwords` from `nltk.corpus`.\n",
    "Go ahead and set `trumpTokens` to a `sub-list` of Trump tokens from `first` to `length of aocTokens`.\n",
    "Then create `FreqDist`s for `trumpTokens` and `aocTokens` and store them in `trumpFreq` and `aocFreq` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "trumpTokens = trumpTokens[ : int(len(aocTokens))]\n",
    "trumpFreq = nltk.FreqDist(trumpTokens)\n",
    "aocFreq = nltk.FreqDist(aocTokens)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"Xen2;1yBSBBi)j~Y?7k0\">stopwords</variable><variable id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</variable><variable id=\"X$P!l]5Lzs,z?%`rx30=\">trumpFreq</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable><variable id=\"`KcUcJ#YWmlEsv^%Id{P\">aocFreq</variable><variable id=\"69-pKS_JOb@!NE-8XbA]\">aocTokens</variable></variables><block type=\"importFrom\" id=\"V#,3E{Xdj3Dx{i%+PX0Y\" x=\"60\" y=\"136\"><field name=\"libraryName\">nltk.corpus</field><field name=\"libraryAlias\" id=\"Xen2;1yBSBBi)j~Y?7k0\">stopwords</field><next><block type=\"variables_set\" id=\",ozm9aVPkB)|1Hm(^H,3\"><field name=\"VAR\" id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</field><value name=\"VALUE\"><block type=\"lists_getSublist\" id=\"Zo}Hkv^Nn+_P9DL+Nmtg\"><mutation at1=\"false\" at2=\"true\"></mutation><field name=\"WHERE1\">FIRST</field><field name=\"WHERE2\">FROM_START</field><value name=\"LIST\"><block type=\"variables_get\" id=\"t)L2CZpuW/,|;FfYKPsm\"><field name=\"VAR\" id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</field></block></value><value name=\"AT2\"><block type=\"lists_length\" id=\"Q,|oTUw/n}@D]=LkgB4:\"><value name=\"VALUE\"><block type=\"variables_get\" id=\"rh%yQG)Ju|:V(]By05)(\"><field name=\"VAR\" id=\"69-pKS_JOb@!NE-8XbA]\">aocTokens</field></block></value></block></value></block></value><next><block type=\"variables_set\" id=\"L6H_P*L)`PU65I)T497@\"><field name=\"VAR\" id=\"X$P!l]5Lzs,z?%`rx30=\">trumpFreq</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"#DM2`=5a5/!a1qCi9R}X\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">FreqDist</field><data>nltk:FreqDist</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"S0=51PFCD~cOG(L+ChIM\"><field name=\"VAR\" id=\"~s1J)yq5==GOG7J(@WTG\">trumpTokens</field></block></value></block></value><next><block type=\"variables_set\" id=\"e%V/6,ab2X{Wi^Ew+lhC\"><field name=\"VAR\" id=\"`KcUcJ#YWmlEsv^%Id{P\">aocFreq</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"{S/%YMa@9VPXi|Yhbq8b\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">FreqDist</field><data>nltk:FreqDist</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"S8PPjSy#P%3G03Q`Sq0T\"><field name=\"VAR\" id=\"69-pKS_JOb@!NE-8XbA]\">aocTokens</field></block></value></block></value></block></next></block></next></block></next></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll remove the stopwords from the frequency distributions.\n",
    "\n",
    "- for each item `j` in list with `stopwords` do `words` using `\"english\"`\n",
    "    - freestyle `del trumpFreq[j]`\n",
    "    - freestyle `del aocFreq[j]`\n",
    "- with `trumpFreq` do `most_common` using `50`\n",
    "\n",
    "*Note: `del` deletes an entry from the FreqDist*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('.', 3133), (',', 2867), ('!', 1469), ('’', 559), ('great', 471), (':', 417), ('...', 377), ('“', 363), ('”', 357), ('&', 326), ('rt', 312), ('-', 285), ('people', 259), ('president', 213), ('…', 212), ('border', 210), ('trump', 205), ('country', 192), ('get', 173), ('(', 166), (')', 162), ('?', 148), ('u', 147), ('democrats', 145), ('news', 144), ('big', 144), ('many', 142), ('vote', 142), ('thank', 141), ('fake', 135), ('new', 129), ('@realdonaldtrump', 122), ('time', 118), ('wall', 117), ('america', 116), ('state', 107), ('military', 104), ('would', 102), ('today', 99), ('never', 98), ('security', 98), ('much', 97), ('house', 95), ('job', 93), ('make', 93), ('done', 92), ('total', 88), ('strong', 88), ('want', 87), ('good', 87)]"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for j in stopwords.words('english'):\n",
    "  del trumpFreq[j]\n",
    "  del aocFreq[j]\n",
    "\n",
    "trumpFreq.most_common(50)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"pK8Q~yy/Q:@KRr:3pEw=\">j</variable><variable id=\"X$P!l]5Lzs,z?%`rx30=\">trumpFreq</variable><variable id=\"Xen2;1yBSBBi)j~Y?7k0\">stopwords</variable></variables><block type=\"controls_forEach\" id=\"F)h_}B(P/oi)u5f8%aX#\" x=\"43\" y=\"157\"><field name=\"VAR\" id=\"pK8Q~yy/Q:@KRr:3pEw=\">j</field><value name=\"LIST\"><block type=\"varDoMethod\" id=\"O;lmtI^s9:3$S.uyiX4.\"><field name=\"VAR\" id=\"Xen2;1yBSBBi)j~Y?7k0\">stopwords</field><field name=\"MEMBER\">words</field><data>stopwords:words</data><value name=\"INPUT\"><block type=\"text\" id=\"X,:-L:}tqcc8!~C;([YO\"><field name=\"TEXT\">english</field></block></value></block></value><statement name=\"DO\"><block type=\"dummyNoOutputCodeBlock\" id=\"A!6B8U|=d^D#eK+(iPx+\"><field name=\"CODE\">del trumpFreq[j]</field><next><block type=\"dummyNoOutputCodeBlock\" id=\"?uW)v48:#/3%hMlfC,2z\"><field name=\"CODE\">del aocFreq[j]</field></block></next></block></statement></block><block type=\"varDoMethod\" id=\"NkyVNxbB8y~[+gUfY?.=\" x=\"-4\" y=\"285\"><field name=\"VAR\" id=\"X$P!l]5Lzs,z?%`rx30=\">trumpFreq</field><field name=\"MEMBER\">most_common</field><data>trumpFreq:most_common</data><value name=\"INPUT\"><block type=\"math_number\" id=\"GQZ=tmI4W(OmYT61k}i.\"><field name=\"NUM\">50</field></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have some punctuation (which isn't in stop lists) but overall we can see some important words like `country`, `military`, and `america`.\n",
    "\n",
    "Display the 50 most common tokens for AOC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('…', 2208), ('.', 2142), (',', 1932), (':', 1634), ('rt', 1167), ('’', 1081), ('!', 707), ('@ocasio2018', 591), ('-', 426), ('“', 340), ('thank', 318), ('&', 233), ('people', 213), ('”', 204), ('\"', 201), ('💜', 188), ('new', 167), ('(', 163), ('us', 159), ('one', 159), ('?', 141), (')', 136), ('today', 132), ('🏽', 130), ('like', 124), ('*', 122), ('+', 121), ('time', 120), ('vote', 111), ('campaign', 108), ('️', 107), ('congress', 103), ('get', 99), ('bronx', 98), ('ny', 98), ('need', 95), ('right', 91), ('queens', 91), ('/', 85), ('make', 85), ('much', 83), ('support', 82), ('help', 81), ('know', 78), ('want', 76), ('day', 75), ('change', 73), ('first', 72), ('14', 72), ('primary', 71)]"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aocFreq.most_common(50)\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"`KcUcJ#YWmlEsv^%Id{P\">aocFreq</variable></variables><block type=\"varDoMethod\" id=\"S(c=?,N:m*cRL+]tle`1\" x=\"-4\" y=\"285\"><field name=\"VAR\" id=\"`KcUcJ#YWmlEsv^%Id{P\">aocFreq</field><field name=\"MEMBER\">most_common</field><data>aocFreq:most_common</data><value name=\"INPUT\"><block type=\"math_number\" id=\"9K%i#7VlYl(0KcFRGU6i\"><field name=\"NUM\">50</field></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------\n",
    "**QUESTION:**\n",
    "\n",
    "Using a few examples, what words overlap with Trump? What words are different?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER: (click here to edit)**\n",
    "\n",
    "*We see some overlapping words with Trump, like `vote` and `want`, as well as regional words like `ny` and `bronx`.*\n",
    "\n",
    "----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conditional distributions\n",
    "\n",
    "Because we have separate Trump and AOC frequency distributions, we can easily create a conditional frequency distribution and tabulate some interesting words.\n",
    "\n",
    "Start by creating a `ConditionalFreqDist` and store it in `cdf`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "cdf = nltk.ConditionalFreqDist()\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"3}CpG*9ZV88If(l`6+lL\">cdf</variable><variable id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</variable></variables><block type=\"variables_set\" id=\"h_^!}yUniw3$0nO[?h:5\" x=\"102\" y=\"184\"><field name=\"VAR\" id=\"3}CpG*9ZV88If(l`6+lL\">cdf</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"Sl1zTF5OtAnMBIyKK^9L\"><field name=\"VAR\" id=\"]HJ?tF]9lV9p9(|h{fmK\">nltk</field><field name=\"MEMBER\">ConditionalFreqDist</field><data>nltk:ConditionalFreqDist</data></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall the last time we made a `ConditionalFreqDist`, we fed it a list of tuples, where the first element was the name of sample and the second element was the word.\n",
    "Here all we need to do is give `cdf` the name of the sample and the `FreqDist` we've already created:\n",
    "\n",
    "- with `cdf` do `update` using a list containing\n",
    "    - freestyle `trump=trumpFreq`\n",
    "    - freestyle `aoc=aocFreq`\n",
    "    \n",
    "Now `tabulate` with the following list of words: `samples=` `'us,people,america,like,love,want,need,vote,change'.split(',')`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       country military  america     like     love     want     need     vote   change \n",
      "  aoc       41        8       40      124       57       76       95      111       73 \n",
      "trump      192      104      116       77       42       87       57      142       14 \n"
     ]
    }
   ],
   "source": [
    "cdf.update(trump=trumpFreq, aoc=aocFreq)\n",
    "\n",
    "cdf.tabulate(samples= ('country,military,america,like,love,want,need,vote,change'.split(',')))\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"3}CpG*9ZV88If(l`6+lL\">cdf</variable></variables><block type=\"varDoMethod\" id=\"#zGq8n;jQH6b8N;AV@it\" x=\"-9\" y=\"188\"><field name=\"VAR\" id=\"3}CpG*9ZV88If(l`6+lL\">cdf</field><field name=\"MEMBER\">update</field><data>cdf:update</data><value name=\"INPUT\"><block type=\"lists_create_with\" id=\"%IR7/gi{)6[8]pf{PKn;\"><mutation items=\"2\"></mutation><value name=\"ADD0\"><block type=\"dummyOutputCodeBlock\" id=\"g/OB%f`uAANY~(OK!pOv\"><field name=\"CODE\">trump=trumpFreq</field></block></value><value name=\"ADD1\"><block type=\"dummyOutputCodeBlock\" id=\"PsF`Ycu!0Mr;ZcaN=n_W\"><field name=\"CODE\">aoc=aocFreq</field></block></value></block></value></block><block type=\"varDoMethod\" id=\"5m:+G5;rJCVD#KU5l~eS\" x=\"1\" y=\"275\"><field name=\"VAR\" id=\"3}CpG*9ZV88If(l`6+lL\">cdf</field><field name=\"MEMBER\">tabulate</field><data>cdf:tabulate</data><value name=\"INPUT\"><block type=\"valueOutputCodeBlock\" id=\"#R7^Q0[Nd,G`gxO8y*En\"><field name=\"CODE\">samples=</field><value name=\"INPUT\"><block type=\"lists_split\" id=\")9HB{gu}Yxls2sd;kW2t\"><mutation mode=\"SPLIT\"></mutation><field name=\"MODE\">SPLIT</field><value name=\"INPUT\"><block type=\"text\" id=\"P;1Puv5Q1J7Jm7x*zbYi\"><field name=\"TEXT\">country,military,america,like,love,want,need,vote,change</field></block></value><value name=\"DELIM\"><shadow type=\"text\" id=\"QG*Y3%.|F^x$c63Ci,Zi\"><field name=\"TEXT\">,</field></shadow></value></block></value></block></value></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see some interesting differences in frequency, and since we've normalized the number of tokens analyzed we can strictly compare these frequencies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------\n",
    "**QUESTION:**\n",
    "\n",
    "What differences seem particularly interesting?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANSWER: (click here to edit)**\n",
    "\n",
    "*Of the words used, Trump is using national identity words more than AOC. This is consistent with identity-based politics.*\n",
    "\n",
    "----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might be wondering why we didn't just convert to probabilities instead.\n",
    "The reason is simple: `tabulate` is designed for integers!\n",
    "However, if you want to create a [probability distribution](https://stats.stackexchange.com/questions/366552/nlp-various-probabilities-estimators-in-nltk) from a frequency distribution, it's as easy as this: `probDist = MLEProbDist(trumpFreq)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "xpython",
   "language": "python",
   "name": "xpython"
  },
  "language_info": {
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}