03_ConnectorSeminar_09022016-Copy2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Discussion 3: Working with Fogel and Engerman's Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from datascience import Table\n",
    "from datascience import *\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plots\n",
    "plots.style.use(\"fivethirtyeight\")\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will explore a smaller version of the Fogel and Engerman dataset, focusing on:\n",
    "\n",
    "/1/Creating a table and visualization for a common qualitative variable V15, Gender\n",
    "\n",
    "/2/Creating a table and visualization for a common quantitative variable V4, Sales by Year \n",
    "\n",
    "One way to demonstrate the usefulness of these activities is to replicate the process with somewhat more involved variables. We look at two. Specifically, V40 is conceptually similar to V15, but tracks whether or not the sale was “guaranteed.” How would you make a table and visualization that depicts this in an appropriate manner? Sometimes the variable seems straight-forward, but presenting it requires some thought. What is an insightful way to depict the number of sales per year?; instead of annual figures, why not present biannual ones, or quinquennial (5 years) or decadal (10 years) figures?\n",
    "\n",
    "/3/Creating a table and visualization for a very uncommon qualitative variable V17, Color\n",
    "\n",
    "Sale records frequently included descriptions of color (listed below), raising a whole host of questions. One question is and how do we, today, know what was considered Griff in the 1800s? \n",
    "\n",
    "V17, skin color:\n",
    "Value    Label \n",
    "1    Negro\n",
    "2    Griff\n",
    "3    Yellow\n",
    "4    Mulatto\n",
    "5    Copper\n",
    "7    Light\n",
    "8    Brown\n",
    "9    Creole\n",
    "11    Dark\n",
    "12    American Negro\n",
    "13    African Negro\n",
    "14  Unknown\n",
    "15  Not Recorded\n",
    "\n",
    "Apparently, at the time, this implied a color lighter than that of those classified as Negro but darker than those classified as Mulatto; here's an article that covers some of this: http://www.uvm.edu/~psearls/johnson.html. Of course, such an answer only pushes the question back, how do we today know the rule for classifying some as Negro and others as Mulatto? These are difficult questions, very difficult questions, and we will discuss how to approach them from a historically minded perspective.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = Table.read_table(\"https://github.com/data-8/history-connector/raw/gh-pages/Data1.csv\")\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "/1/ Gender: The dataset has about 5,000 rows and over 40 variables, many of which are familiar to us from our last discussion. One pattern we noted was that men tended to be represented somewhat more frequently than women in sale records. We can see what pattern the records from New Orleans, a major hub in the slave trade, reveal.\n",
    "\n",
    "We will do this in several steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Selecting V15 from the entire dataset\n",
    "\n",
    "V15_counts = data.group('V15')\n",
    "V15_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Adding labels\n",
    "\n",
    "gender_counts = V15_counts.with_column('Gender', ['Male', 'Female', 'Other'])\n",
    "gender_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Simplification\n",
    "\n",
    "gender_counts = V15_counts.with_column('Gender', ['Male', 'Female', 'Other']).select(['Gender', 'count'])\n",
    "gender_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Visualization\n",
    "#Note, as motivation for these setps, compare: data.group('V15').barh('V15')#\n",
    "\n",
    "gender_counts.barh('Gender')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "/2/ Sales by year: This variable records the year of a sale, and there are no missing values. Histograms are a common way to present data, and we will use them quite a bit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Selecting V4, and presenting it visually\n",
    "\n",
    "viz_salesbyyear = data.select([\"V4\"])\n",
    "viz_salesbyyear.hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#A more detailed view complements the default settings /10 bins/\n",
    "#Note that the dates cover the period 1804-1862\n",
    "\n",
    "viz_salesbyyear = data.select([\"V4\"])\n",
    "viz_salesbyyear.hist(bins=np.arange(1800, 1870, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Another way to present the information is in percent format\n",
    "\n",
    "viz_salesbyyear = data.select([\"V4\"])\n",
    "viz_salesbyyear.hist(bins=np.arange(1800, 1870, 1), normed=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "/3/ Color: We can apply the approach from /1/, and note that the results raise a series of further questions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Selecting V17 from the entire dataset; note the addition of 'show' \n",
    "\n",
    "V17_counts = data.group('V17')\n",
    "V17_counts.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Adding labels\n",
    "\n",
    "color_counts = V17_counts.with_column('Color', ['Negro', 'Griff', 'Yellow', 'Mulatto', 'Copper', 'Black', 'Light', \n",
    "                                                'Brown', 'Creole', 'Quadroon', 'Dark Color', 'American Negro', \n",
    "                                                'African Negro', 'Unknown', 'Not Recorded'])\n",
    "color_counts.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Motivation for the next step, Simplification\n",
    "\n",
    "color_counts.barh('Color')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Simplification \n",
    "\n",
    "color_counts = V17_counts.with_column('Color', ['Negro', 'Griff', 'Yellow', 'Mulatto', 'Copper', 'Black', 'Light', \n",
    "                                                'Brown', 'Creole', 'Quadroon', 'Dark Color', 'American Negro', \n",
    "                                                'African Negro', 'Unknown', 'Not Recorded']).select(['Color','count'])\n",
    "color_counts.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Visualization\n",
    "\n",
    "color_counts.barh('Color')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replication of these results helps reinforce what we have covered. Look over the codebook, and see which variables interest you -- consider exploring the dataset for your project. For now, a suggestion:\n",
    "\n",
    "/a/ Consider V40, and especially think about how would present the information about a \"guarantee of sale,\" for instance in a newspaper or a textbook.\n",
    "\n",
    "/b/ Consider V4, how many bins seem reasonable, what is lost and what is gained by moving away from the default settings?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}