4-Anaconda-IPython-Dataframes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<img src=images/ucsc_banner.png width=500>\n",
    "# Anaconda, IPython, and Dataframes\n",
    "\n",
    "This module covers tools which combine to form an interactive and explorative environment for data science."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/anaconda_logo.png\" width=250>\n",
    "\n",
    "Anaconda is a package manager and collection of popular scientific Python tools. Installing Anaconda is an easy way to setup a data analysis environment from scratch without worrying about conflicting dependencies that may arise from installing so many disparate pieces of software. Anaconda supports Windows, Mac OS X, and Linux."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IPython\n",
    "\n",
    "IPython stands for *Interactive Python*, and is a beefed version of the standard Python interpreter.\n",
    "\n",
    "Type `ipython` in the terminal to open IPython\n",
    "\n",
    "- **M4.Q1** List some *shell* commands that you can use in IPython.\n",
    "- **M4.A1**\n",
    "- **M4.Q2** Why is this useful?\n",
    "- **M4.A2**\n",
    "\n",
    "Type: `a = 'kitties'`  \n",
    "\n",
    "We've now created `a`, a string variable.  But what are some *methods* that you can perform with this string?  In the standard Python intepreter, you'd have a rather difficult time figuring this out without having to look up Python's [string documentation](https://docs.python.org/2/library/string.html). \n",
    "\n",
    "**Terminology**: A *method* is a function that is associated with a particular *object*. An object can be any unique datatype like an *int* or a *str*. \n",
    "\n",
    "Type: `a.` \n",
    "\n",
    "(don't forget the period, or *dot*) and hit *tab*. This returns all of the *dot methods* associated with the *str* object. We see the first method listed is `capitalize`.  Type `a.ca` and hit *tab* again -- this autocompletes the method for us! If we just hit *enter* now, we'll get an error. This is because `capitalize` is a *method*, and all methods and functions are designated in Python by `()`.  So the full command we want is `a.capitalize()`.  \n",
    "\n",
    "- **M4.Q3** What does `a.capitalize()` return? \n",
    "- **M4.A3** \n",
    "- **M4.Q4** How would you *assign* what it returns to a new variable, `b`?\n",
    "- **M4.A4** \n",
    "\n",
    "The reason for the parentheses, is that some methods accept *arguments*. `capitalize` doesn't happen to have any arguments, as it's simply upper-casing the first letter in the string, but what about `count`?\n",
    "\n",
    "Type: `a.count()`\n",
    "\n",
    "This will raise an error: `TypeError: count() takes at least 1 argument (0 given)`.  This tells us that *count* requires at least one argument that goes in the parentheses, but how do we know what that argument is?  \n",
    "\n",
    "Type: `a.count?`\n",
    "\n",
    "```\n",
    "Docstring:\n",
    "S.count(sub[, start[, end]]) -> int\n",
    "\n",
    "Return the number of non-overlapping occurrences of substring sub in\n",
    "string S[start:end].  Optional arguments start and end are interpreted\n",
    "as in slice notation.\n",
    "Type:      builtin_function_or_method\n",
    "```\n",
    "\n",
    "This tells us a lot of information about the `count` method. The first line explains that `count` has a required argument, `sub`, and that it returns an integer (`-> int`).  The rest of the docstring explains `count`'s optional arguments.\n",
    "\n",
    "- **M4.Q5** Provide a valid `count` command.\n",
    "- **M4.A5**\n",
    "- **M4.Q6** Read the docstring carefully and provide a command that uses `count`'s *optional* arguments.\n",
    "- **M4.A6**\n",
    "\n",
    "IPython has a lot more goodies in it, but the above fundamentals give you the basis to explore Python without constantly having to reference documentation. \n",
    "\n",
    "Documentation for IPython can be found at: http://ipython.readthedocs.io/en/stable/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jupyter Notebooks\n",
    "\n",
    "<img src=\"images/jupyter_logo.png\" width=250>\n",
    "\n",
    "Jupyter notebooks used to be called IPython notebooks, but the popularity of notebooks led to the notebook framework being moved into its own project that now supports more languages than just Python.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up remote notebook\n",
    "\n",
    "We want to work on these IPython notebook on our remote server, but notebooks require a GUI to use. We'll launch the notebooks on the remote server, and access them from our local machine using port forwarding.\n",
    "\n",
    "On your remote machine:\n",
    "\n",
    "Type: `screen`\n",
    "\n",
    "This opens a separate terminal so your notebook will stay running even if your SSH instance gets logged out.  You can exit the screen by typing `ctrl+a` then hitting `d`, and resume your screen session by typing `screen -r`.\n",
    "\n",
    "Type: `jupyter notebook --no-browser`\n",
    "\n",
    "From your *local* machine (open a new tab or exit your ssh session):\n",
    "\n",
    "Linux / Mac OSx\n",
    "```\n",
    "ssh -NL 8157:localhost:8888 ubuntu@your-remote-machine-public-dns\n",
    "```\n",
    "\n",
    "You should now be able to point your browser to http://localhost:8157 and see the jupyter notebook startup screen.\n",
    "\n",
    " \n",
    "## Basics\n",
    "\n",
    "The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.\n",
    "\n",
    "Jupyter notebooks are comprised of *cells* that allow a user to write small blocks of code or text and execute them.  You can use the menu at the top to create / delete / arrange cells, or you can click **Help -> Keyboard Shortcuts**. I primarily use only two types of cells: code cells and *Markdown* cells for text formatting.  \n",
    "\n",
    "Besides codifying a methodology in a format that allows for narration to be placed alongside the code used to achieve the result, notebooks are great way to share your research with others as they can download your notebook and run it for themselves.\n",
    "\n",
    "We'll be using notebooks to visualize *dataframes* and execute Python code on those dataframes all in a single environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataframes\n",
    "\n",
    "A *dataframe* is an abstract concept for **(deep breath)** size-mutable \"labeled arrays\" that handle heterogeneous data.  If anyone has used the R programming language, then you're already familiar with what a dataframe is. For those who haven't used R, it simply takes the concept of an *array* of data, and applies hierarchical labeling.\n",
    "\n",
    "## Pandas\n",
    "\n",
    "Pandas is a Python implementation of the the dataframes model with a ton of cool features. I'll let the author himself provide a brief overview of Pandas: https://vimeo.com/59324550\n",
    "\n",
    "I mostly use Pandas for exploratory data science work, some examples of which can be checked out here:\n",
    "\n",
    "**Similarity Comparison Between Two RNA-Seq Pipelines** <br>\n",
    "https://github.com/jvivian/ipython_notebooks/blob/master/RSEM_comparison/RSEM_comparison.ipynb\n",
    "\n",
    "**Fitting a Distribution to Kallisto Bootstraps** <br>\n",
    "https://github.com/jvivian/ipython_notebooks/blob/master/kallisto_boostraps/Kallisto%20Bootstraps.ipynb\n",
    "\n",
    "\n",
    "## Data \n",
    "\n",
    "To play around with Pandas, we'll look at some data from IMDB, the internet movie database.\n",
    "\n",
    "Download cast.csv to the **data/** directory in your forked repo with the following URL: <br>\n",
    "https://drive.google.com/file/d/0ByHO8wS-fc8HTFJpZDE0T3RBcG8/view?usp=sharing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use some CSS styling to make our tables look pretty. This doesn't translate if you view the notebook from Github."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>body {\n",
       "    margin: 0;\n",
       "    font-family: Helvetica;\n",
       "}\n",
       "table.dataframe {\n",
       "    border-collapse: collapse;\n",
       "    border: none;\n",
       "}\n",
       "table.dataframe tr {\n",
       "    border: none;\n",
       "}\n",
       "table.dataframe td, table.dataframe th {\n",
       "    margin: 0;\n",
       "    border: 1px solid white;\n",
       "    padding-left: 0.25em;\n",
       "    padding-right: 0.25em;\n",
       "}\n",
       "table.dataframe th:not(:empty) {\n",
       "    background-color: #fec;\n",
       "    text-align: left;\n",
       "    font-weight: normal;\n",
       "}\n",
       "table.dataframe tr:nth-child(2) th:empty {\n",
       "    border-left: none;\n",
       "    border-right: 1px dashed #888;\n",
       "}\n",
       "table.dataframe td {\n",
       "    border: 2px solid #ccf;\n",
       "    background-color: #f4f4ff;\n",
       "}\n",
       "h3 {\n",
       "    color: white;\n",
       "    background-color: black;\n",
       "    padding: 0.5em;\n",
       "}\n",
       "</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "css = open('style-table.css').read() + open('style-notebook.css').read()\n",
    "HTML('<style>{}</style>'.format(css))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's read in our dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "titles = pd.DataFrame.from_csv('data/titles.csv', index_col=None)\n",
    "cast = pd.DataFrame.from_csv('data/cast.csv', index_col=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two main datatypes in Pandas: DataFrames and Series. It's easiest to think of a single column of a pandas dataframe as a series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(titles.title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two of the most useful pandas methods are `head` and `tail`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The Rising Son</td>\n",
       "      <td>1990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Thousand Plane Raid</td>\n",
       "      <td>1969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Crucea de piatra</td>\n",
       "      <td>1993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Country</td>\n",
       "      <td>2000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Gaiking II</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     title  year\n",
       "0           The Rising Son  1990\n",
       "1  The Thousand Plane Raid  1969\n",
       "2         Crucea de piatra  1993\n",
       "3                  Country  2000\n",
       "4               Gaiking II  2011"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>name</th>\n",
       "      <th>type</th>\n",
       "      <th>character</th>\n",
       "      <th>n</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3499555</th>\n",
       "      <td>Stuttur Frakki</td>\n",
       "      <td>1993</td>\n",
       "      <td>Sveinbj?rg ??rhallsd?ttir</td>\n",
       "      <td>actress</td>\n",
       "      <td>Flugfreyja</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3499556</th>\n",
       "      <td>Foxtrot</td>\n",
       "      <td>1988</td>\n",
       "      <td>Lilja ??risd?ttir</td>\n",
       "      <td>actress</td>\n",
       "      <td>D?ra</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3499557</th>\n",
       "      <td>Niceland (Population. 1.000.002)</td>\n",
       "      <td>2004</td>\n",
       "      <td>Sigr??ur J?na ??risd?ttir</td>\n",
       "      <td>actress</td>\n",
       "      <td>Woman in Bus</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3499558</th>\n",
       "      <td>U.S.S.S.S...</td>\n",
       "      <td>2003</td>\n",
       "      <td>Krist?n Andrea ??r?ard?ttir</td>\n",
       "      <td>actress</td>\n",
       "      <td>Afgr.dama ? bens?nst??</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3499559</th>\n",
       "      <td>Bye Bye Blue Bird</td>\n",
       "      <td>1999</td>\n",
       "      <td>Rosa ? R?gvu</td>\n",
       "      <td>actress</td>\n",
       "      <td>Pensionatv?rtinde</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    title  year                         name  \\\n",
       "3499555                    Stuttur Frakki  1993    Sveinbj?rg ??rhallsd?ttir   \n",
       "3499556                           Foxtrot  1988            Lilja ??risd?ttir   \n",
       "3499557  Niceland (Population. 1.000.002)  2004    Sigr??ur J?na ??risd?ttir   \n",
       "3499558                      U.S.S.S.S...  2003  Krist?n Andrea ??r?ard?ttir   \n",
       "3499559                 Bye Bye Blue Bird  1999                 Rosa ? R?gvu   \n",
       "\n",
       "            type               character     n  \n",
       "3499555  actress              Flugfreyja  24.0  \n",
       "3499556  actress                    D?ra  24.0  \n",
       "3499557  actress            Woman in Bus  26.0  \n",
       "3499558  actress  Afgr.dama ? bens?nst??  17.0  \n",
       "3499559  actress       Pensionatv?rtinde   NaN  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cast.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataframe Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at any column in our dataframe by using array notation `['column_name']` or dot-method notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'title', u'year'], dtype='object')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0             The Rising Son\n",
       "1    The Thousand Plane Raid\n",
       "2           Crucea de piatra\n",
       "3                    Country\n",
       "4                 Gaiking II\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles['title'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0             The Rising Son\n",
       "1    The Thousand Plane Raid\n",
       "2           Crucea de piatra\n",
       "3                    Country\n",
       "4                 Gaiking II\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles.title.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a lot of built-in dataframe methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "title    225616\n",
       "year     225616\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>138980</th>\n",
       "      <td>#1 Serial Killer</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158682</th>\n",
       "      <td>#5</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84531</th>\n",
       "      <td>#50Fathers</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70275</th>\n",
       "      <td>#66</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63318</th>\n",
       "      <td>#73, Shaanthi Nivaasa</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        title  year\n",
       "138980       #1 Serial Killer  2013\n",
       "158682                     #5  2013\n",
       "84531              #50Fathers  2015\n",
       "70275                     #66  2015\n",
       "63318   #73, Shaanthi Nivaasa  2007"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles.sort_values('title').head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>172508</th>\n",
       "      <td>Miss Jerry</td>\n",
       "      <td>1894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120909</th>\n",
       "      <td>Reproduction of the Corbett and Jeffries Fight</td>\n",
       "      <td>1899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92240</th>\n",
       "      <td>Trouble in Hogan's Alley</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19987</th>\n",
       "      <td>Pierrot's Problem, or How to Make a Fat Wife O...</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178104</th>\n",
       "      <td>Soldiers of the Cross</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    title  year\n",
       "172508                                         Miss Jerry  1894\n",
       "120909     Reproduction of the Corbett and Jeffries Fight  1899\n",
       "92240                            Trouble in Hogan's Alley  1900\n",
       "19987   Pierrot's Problem, or How to Make a Fat Wife O...  1900\n",
       "178104                              Soldiers of the Cross  1900"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles.sort_values('year').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conditionals\n",
    "\n",
    "Filtering dataframes can be a bit unintuitive at first, but make sense once you've done it a few times.\n",
    "\n",
    "Say we wanted to look at every movie named **Hamlet**, how would we do that?  You might try something like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         False\n",
       "1         False\n",
       "2         False\n",
       "3         False\n",
       "4         False\n",
       "5         False\n",
       "6         False\n",
       "7         False\n",
       "8         False\n",
       "9         False\n",
       "10        False\n",
       "11        False\n",
       "12        False\n",
       "13        False\n",
       "14        False\n",
       "15        False\n",
       "16        False\n",
       "17        False\n",
       "18        False\n",
       "19        False\n",
       "20        False\n",
       "21        False\n",
       "22        False\n",
       "23        False\n",
       "24        False\n",
       "25        False\n",
       "26        False\n",
       "27        False\n",
       "28        False\n",
       "29        False\n",
       "          ...  \n",
       "225586    False\n",
       "225587    False\n",
       "225588    False\n",
       "225589    False\n",
       "225590    False\n",
       "225591    False\n",
       "225592    False\n",
       "225593    False\n",
       "225594    False\n",
       "225595    False\n",
       "225596    False\n",
       "225597    False\n",
       "225598    False\n",
       "225599    False\n",
       "225600    False\n",
       "225601    False\n",
       "225602    False\n",
       "225603    False\n",
       "225604    False\n",
       "225605    False\n",
       "225606    False\n",
       "225607    False\n",
       "225608    False\n",
       "225609    False\n",
       "225610    False\n",
       "225611    False\n",
       "225612    False\n",
       "225613    False\n",
       "225614    False\n",
       "225615    False\n",
       "Name: title, dtype: bool"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles.title == 'Hamlet'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whoa, what is this? What we've gotten back is a *boolean* list of every title and whether or not it's named **Hamlet**, which we can see a majority of are False.  We can use this to filter our original dataframe by *subsetting it*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5807</th>\n",
       "      <td>Hamlet</td>\n",
       "      <td>1948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27270</th>\n",
       "      <td>Hamlet</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38985</th>\n",
       "      <td>Hamlet</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45362</th>\n",
       "      <td>Hamlet</td>\n",
       "      <td>1910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71187</th>\n",
       "      <td>Hamlet</td>\n",
       "      <td>1954</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        title  year\n",
       "5807   Hamlet  1948\n",
       "27270  Hamlet  2016\n",
       "38985  Hamlet  2015\n",
       "45362  Hamlet  1910\n",
       "71187  Hamlet  1954"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles[titles.title == 'Hamlet'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have multiple conditionals, you need to wrap them with parentheses and combine them with the `&` operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>La momia azteca contra el robot humano</td>\n",
       "      <td>1958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Mavi boncuk</td>\n",
       "      <td>1958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>Perdeu-se um Marido</td>\n",
       "      <td>1957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>Hi no tori</td>\n",
       "      <td>1956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>Quem Sabe, Sabe!</td>\n",
       "      <td>1956</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      title  year\n",
       "38   La momia azteca contra el robot humano  1958\n",
       "43                              Mavi boncuk  1958\n",
       "91                      Perdeu-se um Marido  1957\n",
       "114                              Hi no tori  1956\n",
       "131                        Quem Sabe, Sabe!  1956"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles[(titles.year < 1959) & (titles.year > 1955)].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with Dataframes is *functional*. You can string together many functions and operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>225480</th>\n",
       "      <td>Tischlein, deck dich</td>\n",
       "      <td>1956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64366</th>\n",
       "      <td>Yield to the Night</td>\n",
       "      <td>1956</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       title  year\n",
       "225480  Tischlein, deck dich  1956\n",
       "64366     Yield to the Night  1956"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles[(titles.year < 1959) & (titles.year > 1955)].sort_values('year').head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>172508</th>\n",
       "      <td>Miss Jerry</td>\n",
       "      <td>1894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120909</th>\n",
       "      <td>Reproduction of the Corbett and Jeffries Fight</td>\n",
       "      <td>1899</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 title  year\n",
       "172508                                      Miss Jerry  1894\n",
       "120909  Reproduction of the Corbett and Jeffries Fight  1899"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles[(titles.year < 1959) | (titles.year > 1955)].sort_values('year').head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mutability\n",
    "\n",
    "Another thing that takes some getting used to is Pandas *mutability*.  Pandas prefers never to *mutate*, or change, a dataframe object unless you explicitly tell it to. Instead, it creates a copy and assigns that to the new dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>172508</th>\n",
       "      <td>Miss Jerry</td>\n",
       "      <td>1894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120909</th>\n",
       "      <td>Reproduction of the Corbett and Jeffries Fight</td>\n",
       "      <td>1899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92240</th>\n",
       "      <td>Trouble in Hogan's Alley</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19987</th>\n",
       "      <td>Pierrot's Problem, or How to Make a Fat Wife O...</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178104</th>\n",
       "      <td>Soldiers of the Cross</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    title  year\n",
       "172508                                         Miss Jerry  1894\n",
       "120909     Reproduction of the Corbett and Jeffries Fight  1899\n",
       "92240                            Trouble in Hogan's Alley  1900\n",
       "19987   Pierrot's Problem, or How to Make a Fat Wife O...  1900\n",
       "178104                              Soldiers of the Cross  1900"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted_titles = titles.sort_values('year')\n",
    "sorted_titles.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to force a change into a dataframe, use the `inplace=True` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>172508</th>\n",
       "      <td>Miss Jerry</td>\n",
       "      <td>1894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120909</th>\n",
       "      <td>Reproduction of the Corbett and Jeffries Fight</td>\n",
       "      <td>1899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92240</th>\n",
       "      <td>Trouble in Hogan's Alley</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19987</th>\n",
       "      <td>Pierrot's Problem, or How to Make a Fat Wife O...</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178104</th>\n",
       "      <td>Soldiers of the Cross</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    title  year\n",
       "172508                                         Miss Jerry  1894\n",
       "120909     Reproduction of the Corbett and Jeffries Fight  1899\n",
       "92240                            Trouble in Hogan's Alley  1900\n",
       "19987   Pierrot's Problem, or How to Make a Fat Wife O...  1900\n",
       "178104                              Soldiers of the Cross  1900"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles.sort_values('year', inplace=True)\n",
    "titles.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "Borrowed from the great Brandon Rhodes: http://rhodesmill.org/brandon/\n",
    "\n",
    "Most (if not all) of these exercises can be done in a single line. Be precise! If a question asks \"How many movies...\", then your answer should return a number."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What are the earliest two films listed in the titles dataframe?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many movies have the title \"Hamlet\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many movies are titled \"North by Northwest\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### When was the first movie titled \"Hamlet\" made?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List all of the \"Treasure Island\" movies from earliest to most recent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many movies were made in the year 1950?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many movies were made in the year 1960?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many movies were made from 1950 through 1959?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In what years has a movie titled \"Batman\" been released?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many roles were there in the movie \"Inception\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many roles in the movie \"Inception\" are NOT ranked by an \"n\" value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### But how many roles in the movie \"Inception\" did receive an \"n\" value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display the cast of \"North by Northwest\" in their correct \"n\"-value order, ignoring roles that did not earn a numeric \"n\" value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display the entire cast, in \"n\"-order, of the 1972 film \"Sleuth\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now display the entire cast, in \"n\"-order, of the 2007 version of \"Sleuth\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many roles were credited in the silent 1921 version of Hamlet?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many roles were credited in Branagh’s 1996 Hamlet?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many \"Hamlet\" roles have been listed in all film credits through history?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many people have played an \"Ophelia\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many people have played a role called \"The Dude\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many people have played a role called \"The Stranger\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many roles has Sidney Poitier played throughout his career?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many roles has Judi Dench played?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List the supporting roles (having n=2) played by Cary Grant in the 1940s, in order by year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List the leading roles that Cary Grant played in the 1940s in order by year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many roles were available for actors in the 1950s?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many roles were avilable for actresses in the 1950s?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many leading roles (n=1) were available from the beginning of film history through 1980?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many non-leading roles were available through from the beginning of film history through 1980?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many roles through 1980 were minor enough that they did not warrant a numeric \"n\" rank?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NIH BD2K Center for Big Data in Translational Genomics, UCSC Genomics Institute"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}