Lectures/Week-09-Dictionaries.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# Fall 2018 Status: Just in Time ok\n",
    "\n",
    "# Dictionaries\n",
    "\n",
    "Dictionaries work like the mental model you probably have a dictionary. There are keys, these are the things that will be defined, and there are values or the definitions of the key. Dictionaries in Python are a lot like lists in that they hold collections of data.  However, as we discussed with for loops, they have fundamentally different purposes. As a result, they act in very different ways.  \n",
    "\n",
    "*Important Tips:* Don't try to understand dictionaries like you understand lists.  Yes, they each hold things, but that's where similarites stop.\n",
    "\n",
    "Perhaps the best way to consider dictionaries is as a small database. There are two pieces: unique IDs, and associated records. As noted in the first sentence, we call these keys and values.  \n",
    "\n",
    "## Think of an apartment building\n",
    "\n",
    "This type of data is all around us. In fact, we tend to sort things out this way when given a chance. Take, for example, an apartment. For ease of use, let's call it *your* apartment. \n",
    "\n",
    "Your apartment has a unique address.  First off, the street address tells you which unique buildng you want to find.  There may be several identical buildings, but each has a unique address.\n",
    "\n",
    "Inside that apartment building there are unit IDs.  These may be letters (A, B, etc), labels (Garden, Penthouse, etc), or numbers (6, 303. etc).  Many apartment buildings share the same numbering systems, and this is fine because that apartment belongs to that building.  So the unique combination of street address and unit number can uniquly identify a unit in the same city.\n",
    "\n",
    "Inside these units you can hold whatever you want.  Usually household goods, food, pets, collectables, human bodies, etc.  The idea is that in order to access this unit, you must go into a building and enter a specific ID.  The IDs in that unit must be unique.  \n",
    "\n",
    "You also have to access things in order.  So you can't just directly go into an apartment number.  Even if that apartment door is outside, you still have to find the building.  Imagine saying, \"Hi, I live in New York City and I likein apartment 3.  The party will be at 8pm.\"  Will anyone be at your party?  Not so much.\n",
    "\n",
    "In this analogy, the building is our dictionary.  The street address is our variable name, and the unit number is our key.  You use the key with the variable name to access the contents of that unit.\n",
    "\n",
    "That's pretty much it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mybuilding = {1: ['cat', 'adult human', 'adult human',\n",
    "                  'human child', 'rosy boa', \"Dumeril's boa\"],\n",
    "              2: ['adult human'],\n",
    "              3: ['adult human', 'adult human', 'human child', 'dog?']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If I'm using this to represent my apartment, there are 3 units:  `1`, `2`, and `3`.  Each of these unit labels is actually my key.  See the : in there after each key?  That is separating the key and the value, and the order does matter.  It is always `key: value`.  Meanwhile, each of my values is a list that contains strings of the occupants.\n",
    "\n",
    "As we start covering syntax, I suggest that you write them down or make a small cheat sheet for yourself.  You'll likely want to look it up each time you want to use it until you learn them by heart.\n",
    "\n",
    "# Common patterns that you will want to look up constantly\n",
    "\n",
    "## Accessing values with a key\n",
    "\n",
    "Here's how we can look up the contents:  `dictvariable[key]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['cat', 'adult human', 'adult human', 'human child', 'rosy boa', \"Dumeril's boa\"]\n"
     ]
    }
   ],
   "source": [
    "print(mybuilding[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['adult human']\n"
     ]
    }
   ],
   "source": [
    "print(mybuilding[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['adult human', 'adult human', 'human child', 'dog?']\n"
     ]
    }
   ],
   "source": [
    "print(mybuilding[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if I try to look up a key that isn't in there?  I get a key error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "4",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-2ec4395a1317>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmybuilding\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m4\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m: 4"
     ]
    }
   ],
   "source": [
    "print(mybuilding[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Something to note is that these keys and values can be of any data type, and you can have a combination of them in the same dictionary.  You just need to match how that data type is actually typed in.  Don't forget the quotes if you have a string!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "anotherbuilding = {'Garden': ['Adult human'],\n",
    "                    2: ['Adult human', 'Adult human', 'cat']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding a new key/value pair\n",
    "\n",
    "Say that I leart more about my neighbors and I want to add more things in.  I can add garages as well.\n",
    "\n",
    "The syntax for adding a key/value looks like an assignment statement extending our lookup syntax.\n",
    "\n",
    "`dictvariable[new_key] = new_value`\n",
    "\n",
    "So you place the new key that you want to add in the `[]` and whatever the value is after the `=`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mybuilding['garage 1'] = ['car', 'innumerable crap']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: ['cat', 'adult human', 'adult human', 'human child', 'rosy boa', \"Dumeril's boa\"], 2: ['adult human'], 3: ['adult human', 'adult human', 'human child', 'dog?'], 'garage 1': ['car', 'innumerable crap']}\n"
     ]
    }
   ],
   "source": [
    "print(mybuilding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a closer look at that syntax.  It has the `[]` with the new key on the left side of the `=` statement and only 1 `=`.  So I don't need to completely reassign my dictionary to change it.\n",
    "\n",
    "## Changing the value with a given key\n",
    "\n",
    "Warning!  This syntax is also shared for the reassignment.  If you reuse a key that is in the dictionary, it will overwrite that value without warning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mybuilding['garage 1'] = ['car', 'bike', 'bike', 'innumerable crap']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: ['cat', 'adult human', 'adult human', 'human child', 'rosy boa', \"Dumeril's boa\"], 2: ['adult human'], 3: ['adult human', 'adult human', 'human child', 'dog?'], 'garage 1': ['car', 'bike', 'bike', 'innumerable crap']}\n"
     ]
    }
   ],
   "source": [
    "print(mybuilding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, since our value for each key is in fact a mutator, we can directly reference that object using our access syntax and alter it.\n",
    "\n",
    "Say that the person in apartment 2 has a baby and I want to add that to their record.  Since the data type of that value is a list, I can call .append() to that lookup statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mybuilding[2].append('human child')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: ['cat', 'adult human', 'adult human', 'human child', 'rosy boa', \"Dumeril's boa\"], 2: ['adult human', 'human child'], 3: ['adult human', 'adult human', 'human child', 'dog?'], 'garage 1': ['car', 'bike', 'bike', 'innumerable crap']}\n"
     ]
    }
   ],
   "source": [
    "print(mybuilding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## checking to see if a key is in there\n",
    "\n",
    "You'll often be looping over some data and building up a dictionary in the process.  Oftentimes you'll want to handle things defferently if that key is not already in the dictionary.  For example, let's say I want to add some new people to a unit not already in my dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "5",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-13-808fa0f058c4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmybuilding\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'adult human'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m: 5"
     ]
    }
   ],
   "source": [
    "mybuilding[5].append('adult human')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check this with the `in` keyword that we've seen elsewhere.  This will create a boolean expression that will return `True` if the key does exist in that dictionary, and `False` if it doesn't."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "5 in mybuilding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I can't append something to a key that doesn't already exist, but if I use my assignment statement I'll destroy the data I already have in there.  Here's a fragment showing how we might handle such a situation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "new_member = 'adult human'\n",
    "key = 5\n",
    "\n",
    "if key in mybuilding:\n",
    "    mybuilding[key].append()\n",
    "else:\n",
    "    mybuilding[key] = [new_member]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: ['cat', 'adult human', 'adult human', 'human child', 'rosy boa', \"Dumeril's boa\"], 2: ['adult human', 'human child'], 3: ['adult human', 'adult human', 'human child', 'dog?'], 'garage 1': ['car', 'bike', 'bike', 'innumerable crap'], 5: ['adult human']}\n"
     ]
    }
   ],
   "source": [
    "print(mybuilding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an incredibly common pattern.  You'll also notice that I can reference my key as a variable.  Hint hint, this means you can loop over a set of keys and access the values in turn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the keys and values out separately\n",
    "\n",
    "There are several helpful dictionary methods to use.\n",
    "\n",
    "Warning!  Dictionaries have no actual order.  The following methods will give you lists, which have order, but that order should not be depended on.\n",
    "\n",
    "`mydict.keys()` will give you the keys and `mydict.values()` will give you the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 'garage 1', 5]\n"
     ]
    }
   ],
   "source": [
    "print(list(mybuilding.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['cat', 'adult human', 'adult human', 'human child', 'rosy boa', \"Dumeril's boa\"], ['adult human', 'human child'], ['adult human', 'adult human', 'human child', 'dog?'], ['car', 'bike', 'bike', 'innumerable crap'], ['adult human']]\n"
     ]
    }
   ],
   "source": [
    "print(list(mybuilding.values()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I know they may look like they are in order, but you shouldn't depend on that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## getting the keys and values out together\n",
    "\n",
    "Here's how you can get the pairs of data out such that the pair relationship can be reliably maintained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(1, ['cat', 'adult human', 'adult human', 'human child', 'rosy boa', \"Dumeril's boa\"]), (2, ['adult human', 'human child']), (3, ['adult human', 'adult human', 'human child', 'dog?']), ('garage 1', ['car', 'bike', 'bike', 'innumerable crap']), (5, ['adult human'])]\n"
     ]
    }
   ],
   "source": [
    "print(list(mybuilding.items()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are tuple pairs, and are valuable when looping over them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## looping over dictionaries\n",
    "\n",
    "You may want to loop over the entire thing at once.  Using `.items()` for this is great for when you need both the key and value items out at the same time, and you want to keep your code very tidy.\n",
    "\n",
    "Recall that `.items()` will give you a list that contains tuples with the key and value pairs.  You'll be looping over these tuples, and unpacking those values within the for loop.  This is why you see the `key, value` within the iterable variable space within the for loop declaration line.\n",
    "\n",
    "The key for the pair is stored in `key` and the value is stored in the `value` variable.  This means that you can directly reference them within the loop without having to look up anything else in the process.\n",
    "\n",
    "``` python\n",
    "for key, value in mydict.items():\n",
    "    print(key, value)\n",
    "```\n",
    "\n",
    "This pattern will print out lines that contain the key and the value content for each of the key/value pairs in your dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unit 1 has cat, adult human, adult human, human child, rosy boa, Dumeril's boa\n",
      "Unit 2 has adult human, human child\n",
      "Unit 3 has adult human, adult human, human child, dog?\n",
      "Unit garage 1 has car, bike, bike, innumerable crap\n",
      "Unit 5 has adult human\n"
     ]
    }
   ],
   "source": [
    "for key, value in mybuilding.items():\n",
    "    print(\"Unit\", key, \"has\", \", \".join(value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, sometimes you might want to mess with the keys and only look up a few things.  (massive homework hint) In this case, you can extract out the list of keys, do what you need with it, and then loop through those remaining (LITERALLY DYING OF HINT HERE) keys.\n",
    "\n",
    "Say that I only want the integer value keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rejected! garage 1\n",
      "Remaining keys are: [1, 2, 3, 5]\n",
      "1 has: cat, adult human, adult human, human child, rosy boa, Dumeril's boa\n",
      "2 has: adult human, human child\n",
      "3 has: adult human, adult human, human child, dog?\n",
      "5 has: adult human\n"
     ]
    }
   ],
   "source": [
    "allkeys = mybuilding.keys()\n",
    "\n",
    "justints = []\n",
    "\n",
    "for key in allkeys:\n",
    "    if type(key) == int:\n",
    "        justints.append(key)\n",
    "    else:\n",
    "        print(\"rejected!\", key)\n",
    "        \n",
    "print(\"Remaining keys are:\", justints)\n",
    "\n",
    "for key in justints:\n",
    "    print(key, \"has:\", \", \".join(mybuilding[key]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Counting occurances\n",
    "\n",
    "There are several things we can do to count occurances, but I suggest using the `Counter` object first.\n",
    "\n",
    "Let's revisit last week's example of randomly counting to 100.  Below we've got code that runs the simulation 10,000 times and collects up the number of steps taken for each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The average number of steps to take is: 20.5947\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "# priming all the values\n",
    "count_sum = 0\n",
    "top = 100\n",
    "step_count = 0\n",
    "\n",
    "all_step_counts = []\n",
    "\n",
    "for i in range(10000):\n",
    "    while count_sum < 100:\n",
    "        my_num =  random.randint(0, 10)\n",
    "\n",
    "        count_sum += my_num # sum = sum + my_num\n",
    "        step_count += 1 # step_count = step_count + 1\n",
    "#     print(sum, step_count)\n",
    "    all_step_counts.append(step_count)\n",
    "    # reset!\n",
    "    count_sum = 0\n",
    "    step_count = 0\n",
    "\n",
    "print(\"The average number of steps to take is:\", sum(all_step_counts) / len(all_step_counts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Make a dictionary of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "counts_dict = {} # make an empty dictionary\n",
    "\n",
    "for one_count in all_step_counts:\n",
    "    if one_count not in counts_dict:\n",
    "        counts_dict[one_count] = 1 # start at one if it isn't already there\n",
    "    else:\n",
    "        counts_dict[one_count] += 1 # increment up by one if it isn't already there"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{24: 596, 17: 753, 22: 1156, 27: 158, 18: 1080, 20: 1385, 23: 834, 19: 1274, 21: 1314, 15: 175, 26: 251, 16: 390, 25: 418, 30: 19, 29: 31, 28: 83, 14: 64, 31: 7, 34: 2, 13: 6, 32: 3, 33: 1}\n"
     ]
    }
   ],
   "source": [
    "print(counts_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sorting\n",
    "\n",
    "See how that previous one is in kind of a random order?  We might want to sort it by things.\n",
    "\n",
    "Sorting an unordered list by a key or value and retaining that pair information along the way is rough.\n",
    "\n",
    "Like sorting a column in a speadsheet, but oops, you forgot to tell it to move all the rest of the columns with the rows you are sorting.  \n",
    "\n",
    "## Sorting by key (the easy one)\n",
    "\n",
    "Totally not a problem!  That's the advantage.  You can extract all the keys out, mess around with the order, and then use that content to re-look up the values.\n",
    "\n",
    "There are three steps:\n",
    "\n",
    "1. Get the keys out\n",
    "    * use list(the_dict.keys()) to make them a list\n",
    "2. Sort the keys however you want\n",
    "    * Use your usual list methods for this. (page 345 for the list of list methods)\n",
    "    * Do this on your original list, but it will change it!\n",
    "    * There's no assignment statement.\n",
    "3. Loop through the new list, using the elements you're getting out to look up the values out of the dictionary.\n",
    "\n",
    "You'll have to decide what to do about the things you're getting out.  As a start, you'll want to print out the pairs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13 appeared 6 times\n",
      "14 appeared 64 times\n",
      "15 appeared 175 times\n",
      "16 appeared 390 times\n",
      "17 appeared 753 times\n",
      "18 appeared 1080 times\n",
      "19 appeared 1274 times\n",
      "20 appeared 1385 times\n",
      "21 appeared 1314 times\n",
      "22 appeared 1156 times\n",
      "23 appeared 834 times\n",
      "24 appeared 596 times\n",
      "25 appeared 418 times\n",
      "26 appeared 251 times\n",
      "27 appeared 158 times\n",
      "28 appeared 83 times\n",
      "29 appeared 31 times\n",
      "30 appeared 19 times\n",
      "31 appeared 7 times\n",
      "32 appeared 3 times\n",
      "33 appeared 1 times\n",
      "34 appeared 2 times\n"
     ]
    }
   ],
   "source": [
    "count_keys = list(counts_dict.keys()) # step 1, get the keys\n",
    "\n",
    "count_keys.sort() # step 2, sort it\n",
    "\n",
    "for one_count_key in count_keys:\n",
    "    print(one_count_key, \"appeared\", counts_dict[one_count_key], \"times\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sort by values, the one that sucks\n",
    "\n",
    "So this isn't even a dictionary thing, this is kind of a hack for lists that happens to be useful for dicts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "33 appeared 1 times\n",
      "34 appeared 2 times\n",
      "32 appeared 3 times\n",
      "13 appeared 6 times\n",
      "31 appeared 7 times\n",
      "30 appeared 19 times\n",
      "29 appeared 31 times\n",
      "14 appeared 64 times\n",
      "28 appeared 83 times\n",
      "27 appeared 158 times\n",
      "15 appeared 175 times\n",
      "26 appeared 251 times\n",
      "16 appeared 390 times\n",
      "25 appeared 418 times\n",
      "24 appeared 596 times\n",
      "17 appeared 753 times\n",
      "23 appeared 834 times\n",
      "18 appeared 1080 times\n",
      "22 appeared 1156 times\n",
      "19 appeared 1274 times\n",
      "21 appeared 1314 times\n",
      "20 appeared 1385 times\n"
     ]
    }
   ],
   "source": [
    "def byFreq(pair):\n",
    "    return pair[1] # get the second item out, which is the value\n",
    "\n",
    "pair_list = list(counts_dict.items()) # get all the item pairs\n",
    "\n",
    "pair_list.sort(key = byFreq)\n",
    "\n",
    "for pair in pair_list:\n",
    "    print(pair[0], \"appeared\", pair[1], \"times\") \n",
    "    # this is a list! so I'm using positions, even though ya it totes looks like a key lookup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sup(num):\n",
    "    return num * 1000\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a list of 10,000 data points, but we want to count them up.  We can use `Counter` to do just this.\n",
    "\n",
    "First we'll need to import it to bring the module into our namespace, then we can use the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'h': 2, 'l': 2, 'o': 2, 'e': 1, ' ': 1, 'w': 1, 'd': 1, 'y': 1})\n",
      "[('h', 2), ('l', 2)]\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "print(Counter(\"hello howdy\"))\n",
    "print(Counter(\"hello howdy\").most_common(2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tool takes a sequence (string, list, etc) and counts how many times each item occurs.  This isn't exactly a dictionary, but it acts a lot like one.  Let's apply it to our results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({20: 1365, 19: 1347, 21: 1331, 22: 1117, 18: 1056, 23: 890, 17: 737, 24: 624, 25: 390, 16: 384, 26: 230, 15: 184, 27: 134, 28: 71, 29: 49, 14: 41, 30: 18, 13: 15, 31: 13, 32: 2, 34: 1, 33: 1})\n"
     ]
    }
   ],
   "source": [
    "countedsteps = Counter(all_step_counts)\n",
    "print(countedsteps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34]\n"
     ]
    }
   ],
   "source": [
    "# here are the unique values\n",
    "\n",
    "counts = list(countedsteps.keys()) \n",
    "\n",
    "# now I can sort them\n",
    "\n",
    "counts.sort()\n",
    "\n",
    "print(counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13 : 15\n",
      "14 : 41\n",
      "15 : 184\n",
      "16 : 384\n",
      "17 : 737\n",
      "18 : 1056\n",
      "19 : 1347\n",
      "20 : 1365\n",
      "21 : 1331\n",
      "22 : 1117\n",
      "23 : 890\n",
      "24 : 624\n",
      "25 : 390\n",
      "26 : 230\n",
      "27 : 134\n",
      "28 : 71\n",
      "29 : 49\n",
      "30 : 18\n",
      "31 : 13\n",
      "32 : 2\n",
      "33 : 1\n",
      "34 : 1\n"
     ]
    }
   ],
   "source": [
    "# now I can loop in order\n",
    "\n",
    "for step in counts:\n",
    "    print(step, \":\", countedsteps[step])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13 : X\n",
      "14 : XX\n",
      "15 : XXXXXXXXXXXX\n",
      "16 : XXXXXXXXXXXXXXXXXXXXXXXXX\n",
      "17 : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n",
      "18 : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n",
      "19 : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n",
      "20 : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n",
      "21 : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n",
      "22 : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n",
      "23 : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n",
      "24 : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n",
      "25 : XXXXXXXXXXXXXXXXXXXXXXXXXX\n",
      "26 : XXXXXXXXXXXXXXX\n",
      "27 : XXXXXXXX\n",
      "28 : XXXX\n",
      "29 : XXX\n",
      "30 : X\n",
      "31 : \n",
      "32 : \n",
      "33 : \n",
      "34 : \n"
     ]
    }
   ],
   "source": [
    "# now we can do a janky histogram\n",
    "\n",
    "for step in counts:\n",
    "    print(step, \":\", (countedsteps[step] // 15) * 'X')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, we can filter things on the fly.  Say that we only want to see the values for steps 13 through 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13 : 15\n",
      "14 : 41\n",
      "15 : 184\n",
      "16 : 384\n",
      "17 : 737\n",
      "18 : 1056\n",
      "19 : 1347\n",
      "20 : 1365\n"
     ]
    }
   ],
   "source": [
    "for step in counts:\n",
    "    if step >= 13 and step <= 20:\n",
    "        print(step, ':', countedsteps[step])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if we want to see the most common things, we don't need to do much that's fancy. The counter object comes with some nice things.  Be careful here, because counter really only works well with a large number of things to count.  Sometimes things get weird when you have repeated counts.  It'll give you the 10 most common items, not the items in the 10 most common values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(20, 1365),\n",
       " (19, 1347),\n",
       " (21, 1331),\n",
       " (22, 1117),\n",
       " (18, 1056),\n",
       " (23, 890),\n",
       " (17, 737),\n",
       " (24, 624),\n",
       " (25, 390),\n",
       " (16, 384),\n",
       " (26, 230),\n",
       " (15, 184),\n",
       " (27, 134),\n",
       " (28, 71),\n",
       " (29, 49),\n",
       " (14, 41),\n",
       " (30, 18),\n",
       " (13, 15),\n",
       " (31, 13),\n",
       " (32, 2),\n",
       " (34, 1),\n",
       " (33, 1)]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countedsteps.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a little program we can run to actually grab the items with the top 6 frequencies.  We're doing this by way flipping the dictionary around.  Instead of having a unive value with a count, we want all the unique counts and their values.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(' ', 10), ('h', 8), ('e', 8), ('a', 6), ('o', 5), ('r', 4), ('l', 3), ('y', 3), ('w', 2), ('d', 2), ('t', 2), ('u', 1), ('f', 1), ('i', 1), ('n', 1), ('m', 1), ('s', 1)]\n",
      "{10: [' '], 8: ['h', 'e'], 6: ['a'], 5: ['o'], 4: ['r'], 3: ['l', 'y'], 2: ['w']}\n"
     ]
    }
   ],
   "source": [
    "s = Counter(\"hello howdy how are you fine day more letters here hahahaha\")\n",
    "print(s.most_common())\n",
    "\n",
    "tops = {}\n",
    "\n",
    "howmany = 6\n",
    "\n",
    "pos = 0\n",
    "alls = s.most_common()\n",
    "\n",
    "while len(tops) <= howmany:\n",
    "    content, count = alls[pos]\n",
    "    if count in tops:\n",
    "        tops[count].append(content)\n",
    "    else:\n",
    "        tops[count] = [content]\n",
    "    pos += 1\n",
    "        \n",
    "print(tops)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More crappy state data\n",
    "\n",
    "# heeeey I didn't finish this part 👍\n",
    "\n",
    "Sometimes you'll need to create your own dictionary of things before you can get into the fun stuff. We're going to play with the data that we collected to create a dictionary of what we found.  So that we have the states as our keys and the cities as the values.\n",
    "\n",
    "Remember, this is the code that we previously did.  We could add in the dictionary stuff into this code, but we don't have to.  Instead, we can directly use the list that we create to make the dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "fileio = open('crappystatedata.txt', 'r')\n",
    "\n",
    "text = fileio.read()\n",
    "lines = text.split(\"\\n\")\n",
    "\n",
    "allchunks = []\n",
    "statechunk = []\n",
    "\n",
    "foundfirst = False\n",
    "\n",
    "for line in lines:\n",
    "    if line.endswith(\"[edit]\") and foundfirst == False:\n",
    "#         print(\"This is the first state!\", line)\n",
    "        statechunk.append(line)\n",
    "        foundfirst = True\n",
    "    elif line.endswith(\"[edit]\"):\n",
    "#         print(\"a new state has begun!\", line)\n",
    "        # add completed state\n",
    "        allchunks.append(statechunk)\n",
    "        # reset the chunk\n",
    "        statechunk = [line]\n",
    "    else:\n",
    "        statechunk.append(line)\n",
    "    allchunks.append(statechunk)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "This will be another accumulator pattern, and `{}` represents an empty list.  Remember that we can't add something to a dictionary that doesn't exist, and we can't define the dictionary in our loop, because it'll be erased each time the loop starts over.  So as we've done elsewhere, we need to declare out dictionary outside of and before our for loop through the data.\n",
    "\n",
    "Remember the structure of our allstates list.  This is a list of lists, where the first element of the list is the state name and the remaining elements are all the cities.\n",
    "\n",
    "Let's remember our list position lookups.\n",
    "\n",
    "* \"first element of\" is `list[0]`\n",
    "* \"everything else (after the first)\" is `list[1:]` We're omitting the stop position because there are a variable number of values in each, so when we don't provde a stop position it says \"go until the end\".\n",
    "\n",
    "The state will be our key and the cities will be our value, so let's remind ourselves of the assignment syntax.\n",
    "\n",
    "`dict[state] = cities`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "collegecities = {}\n",
    "\n",
    "for statechunk in allchunks:\n",
    "    state = statechunk[0]\n",
    "    cities = statechunk[1:]\n",
    "    collegecities[state] = cities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "This is great!  Except we've got some junk happening in the state name with the edit.  We can go back in and edit that to make it more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "collegecities = {}\n",
    "\n",
    "for statechunk in allchunks:\n",
    "    state = statechunk[0].replace(\"[edit]\", \"\")\n",
    "    cities = statechunk[1:]\n",
    "    collegecities[state] = cities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['Alabama', 'Alaska', 'Arizona', 'Arkansas', 'California', 'Colorado', 'Connecticut', 'Delaware', 'District of Columbia', 'Florida', 'Georgia', 'Hawaii', 'Idaho', 'Illinois', 'Indiana', 'Iowa', 'Kansas', 'Kentucky', 'Louisiana', 'Maine', 'Maryland', 'Massachusetts', 'Michigan', 'Minnesota', 'Mississippi', 'Missouri', 'Montana', 'Nebraska', 'Nevada', 'New Hampshire', 'New Jersey', 'New Mexico', 'New York', 'North Carolina', 'North Dakota', 'Ohio', 'Oklahoma', 'Oregon', 'Pennsylvania', 'Rhode Island', 'South Carolina', 'South Dakota', 'Tennessee', 'Texas', 'Utah', 'Vermont', 'Virginia', 'Washington', 'West Virginia', 'Wisconsin', 'Wyoming'])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collegecities.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}