3-Modules_Vectors_Plots.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:f900a3dcc1aff2ae107924c323ec232361759abf19bb37274ec5f4c96750142b"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Modules, Vectors and Plotting (Oh my!)\n",
      "\n",
      "__NOTE:__ Before working on the notebook, please go to the 'file' menu, and __save a copy of the notebook!__ This will\n",
      "prevent your file from being overwritten by someone else.\n",
      "\n",
      "## Modules\n",
      "\n",
      "Modules are to Python what appliances are to a kitchen; they make a whole lotta stuff easier to do.  \n",
      "Modules are pre-baked collections of functions, classes and data that fulfill some specific task, e.g. numerical applications, symbolic math, plotting or optimization.\n",
      "\n",
      "To load a module, simply import it"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import sympy # Sympy is used for symbolic math\n",
      "\n",
      "x, y = sympy.symbols('x y')\n",
      "z = sympy.sin(x*y*sin(x))\n",
      "print z.diff(x)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To avoid having to write long names, you can import the module as an alias:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import sympy as sp\n",
      "\n",
      "x, y = sp.symbols('x y')\n",
      "z = sp.sinh(x*y)\n",
      "print z.diff(x) # Still works!"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If you need just a few functions, you can import only those functions:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sympy import symbols, sinh\n",
      "\n",
      "x, y = symbols('x y')\n",
      "z = sinh(x*y)\n",
      "print z.diff(x) # Still works!"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Modules can have submodules. For example, the scipy module has a submodule that has to do with statistical functions:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import scipy.stats as st # Note the alias!\n",
      "\n",
      "# Beta Prime distribution with shape params 2,2\n",
      "bp = st.betaprime(2, 2)\n",
      "print bp.rvs(10) # Generate some random numbers"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The remainder of this session goes through the most commonly used modules for scientific work. Below are the modules I typically load when working."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#import numpy as np # Numerics (vectors, matrices, ...)\n",
      "#import matplotlib.pyplot as plt # Plotting functions\n",
      "#import seaborn as sns # Pretty plots!\n",
      "#import pandas as pd # Structured data, .csv files, time series"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Additional modules that may be worth knowing about:\n",
      " - Scipy (Huge collection of miscellanous scientific functions, statistics, regression etc.)\n",
      " - Basemap (Nice plotting for maps)\n",
      " - Pygrib (Geographical data handling)\n",
      " - BeautifulSoup (Parsing .xml files easily)\n",
      " - PyTables (Used for large dataset that can't be kept in memory)\n",
      " - PyOmo (Modeling library)\n",
      " - Scikit-learn (Machine learning library)\n",
      " - NetworkX (Network/Graph data library)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Vectors and linear algebra with Numpy"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Numpy implements vector, matrix and tensor structures in Python. All calculations are implemented in high-performance C and FORTRAN libraries, which means calculations are a lot faster than writing the loops out in pure Python.\n",
      "This is typical for Python libraries; even though you're writing in Python, the calculations in the background may take place in another language in order to make them faster."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "\n",
      "a = [1,2,3,4,5,6,7,8,9]\n",
      "c = np.array(a)\n",
      "print c*3\n",
      "print c**2\n",
      "print c*c # Equivalent to c.*c in MATLAB\n",
      "print np.dot(c,c) #For vectors, orientation doesn't matter in dot products.\n",
      "print c.dot(c) # Some functions are available on the vectors as well"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Matrices are implemented via lists-of-lists."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "A = np.array([[1,2,3],[4,5,6]])\n",
      "b = np.array([7,8,9])\n",
      "print A\n",
      "print A.dot(b)\n",
      "print A.T # Transposition\n",
      "print A.T.dot(b) #Gives error due to orientation"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Numpy has most of the \"standard\" library of linear algebra functions. (More functions are available in the SciPy module)\n",
      "\n",
      "__EX:__ Generate a random 4x4 matrix A (Hint: np.random. )\n",
      "\n",
      "Find the eigenvalues and eigenvectors of that matrix (Hint: np.linalg. )\n",
      "\n",
      "Generate a random 4x4x4 matrix B.\n",
      "\n",
      "Use Einstein summation (np.linalg.einsum?) to calculate the sum\n",
      "$$\\sum_{ij} A_{ij}, B_{ijk} = c_k$$\n",
      "\n",
      "Solve the system $ A \\vec{x} = c $ (Hint: np.linalg.solve)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Write here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Plotting data using Matplotlib\n",
      "\n",
      "Matplotlib handles everything to do with plots."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import matplotlib.pyplot as plt\n",
      "# Tell matplotlib that we want to have plots appear in the notebook\n",
      "%matplotlib inline "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "x = np.linspace(-10,10,201) #Generate evenly spaced points from -10 to 10\n",
      "plt.plot(x, np.sin(x)) # A plot!\n",
      "plt.plot(x, np.cos(x)) # Another line on the same plot!\n",
      "# (In MATLAB parlance, hold is on by default)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Since this workshop is run on a shared server,\n",
      "# please close down your figures when done\n",
      "# to free up RAM - run this box every once in a while\n",
      "plt.close('all')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Matplotlib supports subplots, legends and labels ala MATLAB. Plot parameters can be passed via keyword:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ax1 = plt.subplot(211)\n",
      "plt.plot(x, np.cos(np.arctan(x/3)*x), label='Wobbly state')\n",
      "plt.legend(frameon=True, fancybox=True)\n",
      "plt.ylabel('Swing state')\n",
      "ax2 = plt.subplot(212)\n",
      "plt.plot(x, np.sin(x),\n",
      "         label='Hi2', color='red',\n",
      "         linestyle='--', linewidth=4)\n",
      "plt.xlabel('Time delta [s]')\n",
      "plt.ylabel('Swing state, pendulum 2')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "2-D plots are available as well:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "x = np.linspace(-5, 5, 101)\n",
      "y = x.copy()\n",
      "X, Y = np.meshgrid(x, y) # 2d arrays\n",
      "plt.figure(figsize=(8, 6)) #Make a figure that will be 8 by 6 inches\n",
      "# Contour plots allow for custom color maps, ranges, etc.\n",
      "plt.contourf(X, Y, np.sin(X)*np.sin(Y), 21, # use 21 levels\n",
      "             vmin=-0.7, vmax=0.7, extend='both',\n",
      "             cmap=plt.cm.RdBu_r)\n",
      "cb = plt.colorbar()\n",
      "cb.set_label('Anomaly [cm]')\n",
      "plt.draw() #Update the colorbar label"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Once you're happy with a plot, you can save it to disk:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plt.savefig('myfig.pdf') # Move this to the cell above; save commands must be in the same cell as the plotting commands."
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The standard matplotlib colors and layout are pretty basic. The seaborn package has some prettier standard settings, and some extra functions that are useful for data visualization. Just importing the package is enough to add the new settings:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import seaborn as sns\n",
      "sns.set_style('ticks') # I like the \"ticks\" style for plotting\n",
      "\n",
      "x = np.linspace(-10,10,201)\n",
      "plt.plot(x, np.sin(x)) # Now with more pretty!\n",
      "plt.plot(x, np.cos(x))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "__EX:__ Making plots that are both clear and concise is necessary to communicating what you want with the data.\n",
      "\n",
      "The plot below shows some temperature data with an increase over time, but the author clearly hasn't thought about how best to show this. Clean up the figure in some of the following ways:\n",
      "\n",
      "- Look at the function sns.xkcd_palette, and the associated dictionary sns.xkcd_rgb, and use colors from this dictionary for the tasks below.\n",
      "- Switch from using scatter to using plot with no line, give the points a nicer color, and use the alpha keyword to make the points slightly transparent.\n",
      "- Use sns.regplot to plot a second-order regression line for the data. Remember to select nice colors! (Hint: Look at sns.regplot? for the parameters)\n",
      "- Add labels to the axes\n",
      "- Add labels to each of the plot commands, and add a legend to the plot\n",
      "- Use plt.grid to add a grid. Set the grid to be transparent by using the alpha parameter.\n",
      "- Use sns.despine to get rid of the upper and right-hand axes"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "x = np.arange(2*1980, 2*2014)/2.0\n",
      "y = 20 + (x - 1950)**0.1*(1 + 0.1*np.random.random(x.shape)) \\\n",
      "+ 0.1*np.random.normal(x.shape)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 116
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plt.scatter(x, y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 117,
       "text": [
        "<matplotlib.collections.PathCollection at 0x7f373ef84690>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEJCAYAAABohnsfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHr5JREFUeJzt3X+QHPV55/H38iMI4SpyggA+1j7IwTxgTrHLOesuQiCk\nJZISnGCCA1hCZCXiiI1FUuHiE6ack4QLKKPgMtjy7kE4CysgneWCxLDGybJQqziSrSQV2zjhnpMS\n6SzZyMK6iF/Hoo2Y+6N7pdFodqZnpqd/zedVtaVRT39nnv7ubD/z/dHf7imXy4iIiERxUtoBiIhI\nfihpiIhIZEoaIiISmZKGiIhEpqQhIiKR5S5pmNmatGOoJYtxKaZoFFN0WYxLMUUTV0w9jabcmtn9\nwBzgFOA+4CBwDzABvAksdfdDFftfBWwBfhBuetHdf8/M3gNsJEhUL4flDjcbsJmV3b2n2XKdlsW4\nFFM0iim6LMalmKKJK6a6LQ0zmwdc5u6zgUXAg8ADwHJ3nw9sA1bUKPqCu88Lf34v3HY38AV3vxLY\nBSxvN3gREUlWo+6prcAN4eNXgenAAeDscNsM4JUa5Wpls7nA18PHTwNXNxWpiIik7pR6T7r7EYIu\nKIBbgWHgXmDMzA4RdFWtqipWBt5nZn9OkFTWuvtzwBnuPhHu8wrw7ngOQUREktJwTAPAzK4F7gQW\nAk8Cf+Tu281sHbDH3ddX7PtvgcvdfYuZ/TzwAnARsM/dzw33uQh4zN0vbyZYMzsNGA9f70gzZROw\nG7gw7SCqKKZoFFN0WYxLMTV2MsGwwDR3f7udF6rb0gAws4XAXcBCd3/NzGa6+/bw6RFgCXA0abj7\njwkGwnH3fzaz/cD5wBtmdloY8PnAj1uI90Phv7taKJuE3WkHUINiikYxRZfFuBRTNB8CvtXOC9RN\nGmZ2JrAOmF8xQ2q/mV3q7i8Bs4CdVWUWAxe7+1ozOwc4B/gR8BzwUeBx4Hrg2RbifRng8ccf57zz\nzmuhuIhI99m/fz9LliyB8BzajkYtjRuBs4AtZja5bSXwiJlNEIxpLAcws01AP8Fg9xNm9i2CJtGA\nu0+Y2WrgK2a2AtgDPNZCvEcAzjvvPHp7e1soLiLS1dru1m80EP4w8HCNp+bU2Pdj4cO3gV+v8fx+\nYEELMYqISEbk7opwERFJj5KGiIhEpqQhIiKRKWmIiEhkShoiIhKZkoaIiESmpCEiIpEpaYiISGRK\nGiIiEpmShoiIRNZwlVsRkSwYHx9nw4ZRAPr7+5g2bVrKEXUnJQ0Rybzx8XEWLXqCsbGlAGzevJFv\nfnOxEkcK1D0lIpkxPj7O0NAwQ0PDjI+PH92+YcNomDBOBU5lbOzmo60OSZZaGiKSCVltTahb7Hhq\naYhIJtRrTfT39zF37kbgMHCYuXP/lP7+vo7HNJnIBgYWMDCwgEWLnjiuBdSNlDREJPOmTZvGN7+5\nmMHBEQYHRxJrgahb7ETqnhKRTOjv72Pz5o2Mjd0MELYmFh99ftq0adx22zVphSchJQ0ROUEa/fiT\nrYkNG0bC901/PKNRIutGShoiBRHXiT7NAemstSaymMjSpqQhUgBxnuiP78cn7McfydTJPElZS2Rp\n00C4SAFowFaSoqQhIsdJa3qr5IOShkgBxHmiT2t6q+SDxjRECiDuAdtO9ePr6ur8U9IQKYisD9hm\ndZkQaY66p0QkERqsLwYlDRERiUxJQ0QSoVlZxdBwTMPM7gfmhPveBxwE7gEmgDeBpe5+qEa504Ef\nAGvd/StmtgH4YFgeYJ27fyOOgxCR7NPV1cVQN2mY2TzgMnefbWYzgO8CPwEWu/tOM/sUsAL4bI3i\nn+ZYggAoA3cqUYgcr5tmFGV9sB666/fRikYtja3AjvDxq8B04ABwNrATmAG8VF3IzC4BLgGGgZ6K\np3qq9xXpZppRlC36fTRWN2m4+xGCLiiAWwmSwL3AmJkdImhJrKpRdB3wCWAZQQtj0kozu4Mg8ax0\n94M1yop0Da3zlL7KlsXExIR+Hw1EGgg3s2sJEsDtwHrgOne/BNgGDFTtewuw1d1/SNCymGxdbARW\nuXsfQTfXmjgOQESkVdV35nvooR2NC3W5KAPhC4G7gIXu/pqZzXT37eHTI8ASgkQy6VeBnzez3wB6\ngbfNbK+7P1+xz9PAlxq87xpgdeQjEckh3a8hXdUtvV277uTiiz/Pzp2/DxTy97HbzKq3rXX3NVFf\noNFA+JkEXU3zK2ZI7TezS939JWAWwdjGUe5+U0X51cBud3/ezL4WBvcicCXwYr33Dg/iuAMxswuA\n3Y0PSyQfNKMoa6Zx++0Xceqphf19XOjue9p5gUYtjRuBs4AtFdlpJfCImU0QjGksBzCzTcAyd5/q\nrutfBL5sZm8ArxN0d4l0vTzMKCqqWi29j3+8cIkiVj3lcrnxXhkx2dIYHR2lt7c37XBEpAC6YYrt\nvn376OvrgwRaGiIihaaWXnOUNESm0A3fQEWapaQhUoMu8hKpTQsWitSgZbxFalPSEBGRyJQ0RGrQ\nMt4itWlMQ6QGXXQnUpuShsgUNBVT5ETqnhIRkcjU0hCRrqNrcFqnpCEiXUXX4LRH3VMi0lV0DU57\nlDRERCQyJQ0RyaTx8XGGhoYZGhpmfHyqOy40T9fgtEdjGiKSOe2OO9Qb6NY1OO1R0hCRzKm+DWsw\n7jAS6bqZKAlH1+C0Tt1TIlIoGujuLCUNEckcjTtkl5KGiGTO5LjD4OAIg4MjTY1nKOF0lsY0RCST\nWh130EB3ZylpSK5pOQipRQPdnaOkIbml5SBEkqcxDcktzZIRSZ5aGiItULeYdCu1NCS30polM9kt\nNjCwgIGBBSxa9ESsy1yIZJmShuRWO9My26FuMelm6p6SXNMsGZFkqaUh0iRdPCbdTC0NkSbp4rH6\nNEmg2BomDTO7H5gT7nsfcBC4B5gA3gSWuvuhGuVOB34A3O3uj5nZe4CNBK2bl8Nyh+M6EJEk5bFb\nLImTua6dKb663VNmNg+4zN1nA4uAB4EHgOXuPh/YBqyYovinCRJMOfz/3cAX3P1KYBewvP3wRSSK\npGZ8aZJA8TUa09gK3BA+fhWYDhwAzg63zQBeqS5kZpcAlwDDQE+4eS7w9fDx08DVLUctIk3p9pN5\np+4C2I3qdk+5+xGCLiiAWwmSwL3AmJkdImhJrKpRdB3wCWAZx1oaZ7j7RPj4FeDd7YUuRVX0PvE4\njy9rddXf38fmzRsZG7sZIJwksDjVmNRlFq9IA+Fmdi1BAlgIPAlc5+7bzWwdMACsr9j3FmCru//Q\nzHo41tKoVGtb9XuuAVZHiU+Koyh/4FOdzOM8vmZeK6mTeRYnCbRzF8AC2m1m1dvWuvuayK9QLpfr\n/pRKpYWlUuk7pVLpZ8P//6TiuQWlUumxqv03l0qlHaVSaXupVNpbKpV2lkqlvlKp9E+lUmlauM/c\nUqm0pdF714jlglKpVN67d29Zimlw8JkyHC5DOfx5uzw4+Ewi7/3WW2+VBwefKQ8OPlN+66232nqd\nuXMfDY/jcHnu3EePvl6zx1cvpjhfq8jS/Exlxd69e8ulUqlcKpUuKDd53q3+aTQQfiZBV9M1FTOk\n9pvZpeHjWcDOyjLufpO7z3L3XwL+BPiMu48CzwHXh7tdDzwbObOJdFijgeJm+sTjGj+Ie/B6csbX\nbbddk/q3/yTpupp4NRoIvxE4C9hiZi+Y2QvASuCR8PEHgIcAzGyTmdX7JK4GfsvMtgI/CzzWdvSS\nqCQGE9P6A693oo/z5N3M8TVKPjoZRpPWcjOF1W5TJckfdU+lp16XSyfeK+lulHpdGK10AwV19XYZ\n3j6hrqIeX5T37dYuJ2lOnN1TqSeCZn6UNNJT9H7heif6Vo49jpN5o+QjElWcSUPLiIhQf9ZPKzOP\n4rhiPIszkUR6yuVy470ywswuAHaPjo7S29ubdji51Oq8/mPTO4+dOLupbzhr10OINGPfvn309fUB\nXOjue9p5LbU0ukg71wh0+7ferKw1peQlaVPSyIgkTgbtXuSUlRNnt0rzwkclK5mk+2lkgG4fKlGk\ntX5UJz+fWhMqf5Q0MiCpk4Hm9UsrOvX51JelfFLSKKCpvr1l9SKnInzbLPKFj53S7Svv5la7c3aT\n/CnqdRpxzsdP8iK8OOQt3lqKfuFjp64XKfq1P1mii/sKljTK5fhOBnn7Q0w63k6cdPNW563oRL3p\n4sXk6OK+AtLMpM4ryrLraaj3+Wx1ZlW3T+POrXazTpI/RW5pxCVv396SjLdTLYK81XmcitC92A3U\n0pAp5e3bW97iraUIx9Aq3eCo+yhpFECt7oE8/dEmFW8n716XtzoXaZWSRs6pnz66bm4RxKnyS8pN\nN12euXuCS2cpaeScugeaoxZBe2p9SfmzP/sNNm9WIu4WShpdrp01hZJaj0jrHmVHrS8pmzdn40uK\nPifJUNLIuXb66dvp2kqqW0zdb/mXxMlcn5MEtTv9KskfTbmtrdULr9qZgtrKLVCTjlHi1+z04qSm\n5OpzUp+m3Mpxst5Pr2+BxdHsZIJOjrlVtmAmJibafj2JqN2sk+SPWhrxaueitGbKtvMtsJsvnGtX\nGutUVev8BZVBC+aKK4bKV1zxsD4nU1BLI0PyPPjWzhTUpKavappsa7LSuuvUtTHVLZi/+qtlPPTQ\nMIsX63PSce1mnSR/stbS0BIK0ai1kLws9fFrkcj0xdnS0P002qD7AUST1ft4SDImx9xuu+2a2H7v\nRbu3SJ6oe0oSkfXB+qLp5JIpWaBuy/QoabSh6H+Ykl/dcFLVF5F0KGm0oRv+MCW/dFKVTlDSaJP+\nMLMlz7PZRPKgYdIws/uBOeG+9wEHgXuACeBNYKm7H6rYfzqwATgHmAZ8xt2HzWwD8MGwPMA6d/9G\nbEciXS8r00xFiqzu7Ckzmwdc5u6zgUXAg8ADwHJ3nw9sA1ZUFfswsMPdrwJuAD4Xbi8Dd7r7vPBH\nCUNipdlsIp3XqKWxFdgRPn4VmA4cAM4GdgIzgJcqC7j7Vyv++15gb8X/e9oJtpuom0VEsqhu0nD3\nIwRdUAC3AsPAvcCYmR0i6GpaVausmW0DzgcqO/xXmtkdBIlnpbsfrFW22+W1myXtRFeE2Wxp16FI\nI5EGws3sWmAZsBB4ErjO3beb2TpgAFhfXcbdZ5vZ+4HHgfcDG4Gfuvv3zWwVsAa4vc57rgFWN3U0\nBZHHGytlIdHlfTZbFupQCm+3mVVvW+vua6K+QJSB8IXAXcBCd3/NzGa6+/bw6RFgCRVJw8x+ETjg\n7nvd/XtmdoqZne3uz1e87NPAl+q9b3gQxx2ImV0A7G54VJK4rCS6PM9my0odSqFd6O572nmBRgPh\nZwLrgGsqZkjtN7NLw8ezCMY2Kl0B3BGWPxc4AzhoZl8zs5nhPlcCL7YTeJGluUTC+Pg4Q0PDDA0N\nMz4+nsh7ikh+NGpp3AicBWypaNKsBB4xswmCMY3lAGa2CegHhoBHzWwrcDrwCXcvm9kXgS+b2RvA\n6wTdXVJDWt0s7XSPFGE8IW2qQ8mDnnK5nHYMkU12T42OjtLb25t2OLkRdXB1aGiYgYEFTHaPwGEG\nB6N3j2gQt32qQ+mEffv20dfXBzF0T+mK8IJLcnA1z+MJWaE6lKzT0ugF18wFb1puWkQaUUtDjsr7\nlFVpnbrFJColjYJrdnA1qe4RnaTSVVn/N910OR/5yJO6PkQiUdIouHZaD506sSc5zqLkdKLq+n/g\ngdXs2rUWXR8iUShpdIFWWg+dPLEndRGbrrCurbr+d+2alW5AkisaCJeairBibBGOIRnzufjiz6MJ\nEBKFkoYkTrO00nVi/X+NHTs+zuDgCIODI2qNSV26uE9qOta1c2wAPc6TSRJjDZ0+hjzTWE93ifPi\nPiUNmVIRTixFOAaRdumKcElEEa5OzsIxKHFJkShpiHSQZnBJ0WggXKSDNINLikZJQ0REIlPSEOkg\nTS+WotGYhuROngaWtQikFI2ShuRKHgeWszCDSyQu6p6SXNHAski6lDRERCQyJQ3JFQ0si6RLYxo5\nlKeB4LhpYFkkXUoaOZPHgeC4aWBZJD3qnsoZDQSLSJrU0pCu1c3dfCKtUksjZzQQHI/Jbr6BgQUM\nDCxg0aInGB8fTzsskcxT0siZyYFg3WWtPermE2mNuqdySAPBIpIWtTSkK6mbT6Q1DVsaZnY/MCfc\n9z7gIHAPMAG8CSx190MV+08HNgDnANOAz7j7sJm9B9hIkKheDssdjvVoRCLS9R4iranb0jCzecBl\n7j4bWAQ8CDwALHf3+cA2YEVVsQ8DO9z9KuAG4HPh9ruBL7j7lcAuYHlcByHSisluvttuu0YJQySi\nRi2NrcCO8PGrwHTgAHA2sBOYAbxUWcDdv1rx3/cCe8PHc4HfCR8/DfwhMNRq4CIikry6ScPdjxB0\nQQHcCgwD9wJjZnaIoKtqVa2yZrYNOB+YHLE9w90nwsevAO9uL3QREUlapNlTZnYtsAxYCDwJXOfu\n281sHTAArK8u4+6zzez9wOPA+6ue7onwnmuA1VHiExGRSHabWfW2te6+JuoLRBkIXwjcBSx099fM\nbKa7bw+fHgGWUJE0zOwXgQPuvtfdv2dmp5jZzwFvmNlp7v42QQvkx/XeNzyI4w7EzC4Adkc9OBER\nOc6F7r6nnRdoNBB+JrAOuKZihtR+M7s0fDyLYGyj0hXAHWH5c4F3AT8FngM+Gu5zPfBsO4FLto2P\njzM0NMzQ0LCutBYpkEYtjRuBs4AtFU2alcAjZjZBMKaxHMDMNgH9BIPbj5rZVuB04HfdvWxmq4Gv\nmNkKYA/wWLyHIlmhlXhFiqunXC6nHUNkk91To6Oj9Pb2ph2OTGFoaJiBgQUES3QAHGZwcERXsYuk\nZN++ffT19UGnu6dEREQqKWlI7LREh0hxacFCiZ2W6BApLiUN6QitxCtSTEoaKdKd40Qkb5Q0YtRM\nEtC0VBHJIw2Ex6TZ24fqznEikkdKGjFREhCRbqCkkZJmp6VqWQ4RyQKNacSkv7+PzZs3MjZ2M0CY\nBBZPuX8z01I1/iEiWaGkEZNWrk2IOi31+K4vwq4vLcshIslT0oiRrk0QkaLTmEYOaFkOEckKtTRy\nQMtyiEhWKGnkhLq+RCQL1D0lIiKRKWmIiEhkShoiIhKZkoaIiESmpCEiIpFp9lSCdP8MEck7JY2E\naP0oESkCdU8lREuni0gRKGmIiEhkShoJ0fpRIlIEGtNIiNaPEpEiUNJIkNaPEpG8U/eUiIhE1rCl\nYWb3A3PCfe8DDgL3ABPAm8BSdz9Ur4y7P2VmG4APhuUB1rn7N2I6DhERSUDdpGFm84DL3H22mc0A\nvgv8BFjs7jvN7FPACuCzdcr8PfAUUAbuVKIQEcmvRi2NrcCO8PGrwHTgAHA2sBOYAbzUoMwZZjbZ\nDdbTbsAiIpKeuknD3Y8QdEEB3AoMA/cCY2Z2iKCraVWjMu7+jpkBrDSzOwgSz0p3P0iBadkQESma\nSAPhZnYtsAy4HVgPXOfulwDbgIE6ZZYDK8NNG4FV7t5H0M21pq3IM25y2ZCBgQUMDCxg0aInGB8f\nTzssEZG2RBkIXwjcBSx099fMbKa7bw+fHgGWECSS6jKfAha5++sA7v58xS5PA19q8L5rgNURjyNz\njl82hHDZkBFNuRWRNO0Oe30qrXX3NVFfoNFA+JnAOmB+xQyp/WZ2qbu/BMwiGNtoVAYz+1oY3IvA\nlcCL9d47PIjjDsTMLgB2NzwqERGp5UJ339POCzRqadwInAVsqchOK4FHzGyCYExjOYCZbSLowqpV\n5hbgi8CXzewN4PVw38Lq7+9j8+aNjI3dDBAuG7I45ahERNrTUy6X044hssmWxujoKL29vWmH05AG\nwkUkC/bt20dfXx8k0NKQNmjZEBEpGi0jIiIikSlpiIhIZEoaIiISmZKGiIhEpqQhIiKRKWmIiEhk\nShoiIhKZkoaIiESmpCEiIpEpaYiISGRKGiIiEpmShoiIRKakISIikSlpiIhIZEoaIiISmZKGiIhE\npqQhIiKRKWmIiEhkShoiIhKZkoaIiESmpCEiIpEpaYiISGRKGiIiEpmShoiIRKakISIikSlpiIhI\nZEoaIiIS2SmNdjCz+4E54b73AQeBe4AJ4E1gqbsfqlfG3Z8ys/cAGwkS1cthucMxHouIiHRY3ZaG\nmc0DLnP32cAi4EHgAWC5u88HtgErGpT5fPjU3cAX3P1KYBewPM4DERGRzmvUPbUVuCF8/CowHTgA\nnB1umwG80qDMGWZ2EjAX+Hq4/Wng6tbDFhGRNNTtnnL3IwRdUAC3AsPAvcCYmR0i6Kpa1aiMu79j\nZme4+0S4/RXg3fEcgoiIJKXhmAaAmV0LLAMWAk8C17n7djNbBwwA66cosxz45Rov2dNivCcD7N+/\nv8XiIiLdp+KceXK7rxVlIHwhcBew0N1fM7OZ7r49fHoEWEJV0gjLfApY5O6vh5vfMLPT3P1t4Hzg\nxw3edw2wutZzS5YsaRS2iIicaJeZVW9b6+5ror5A3aRhZmcC64D5FTOk9pvZpe7+EjAL2BmhDMBz\nwEeBx4HrgWfrvXd4EMcdiJmdBowDFwFH6pVPwW7gwrSDqKKYolFM0WUxLsXU2MkEE5CmhV/cW9ZT\nLpenfNLMfofg2/7/rtj83wim3k4QjGksD1sgmwi6sG6pUWYp8K/AV4BpwB5gWTj+0RQzK7t7q91b\nHZPFuBRTNIopuizGpZiiiSumRgPhDwMP13hqTo19PxY+nKoMwIKmohMRkUzRFeEiIhKZkoaIiESW\nx6SxNu0AppDFuBRTNIopuizGpZiiiSWmugPhIiIilfLY0hARkZQoaYiISGRKGiIiEpmShoiIRKak\nISIikUVa5TYpZvYLwFPA59x9vZldQnB1+TsEy5IMAB8A/rii2PuAjwAGfIZgfRWAEXe/N4mY3P2I\nmd1DcM+Qk4Cn3H2dmZ0KbADeS7BW1jJ3351yTP0EN8T6p/Dlkq6nFQRL5h8O932yU/UUQ1z9dKau\nqu+G+bfUuKulmS0Bfj+M9WF3/x8d/Ey1E1M/6dbTDGAT8Jq7/2ZYNu16qhVTP+nW043AHQS/u1F3\n/3Qr9ZSZloaZTSe4K+BfAJPzgD8L3OPuVwH7gBvc/e/cfZ67zyNIFv8YrrpbBjZNPhfTLyNSTGb2\nH4Cr3H0OcDmwzMzOBRYD/9fdryC4Re59GYipDGxOqZ7OAf4LwQe8D/ikmU2jA/UUU1ydqKtad8Nc\nS9VdLc3sDOCPwniuAv7AzP4NnflMtRtTavUU7r4eeL7qJVKrpzoxpfl5Op3gs9/n7r8EXG1ml9JC\nPWUmaQBvAx8GflKx7SJgR/j4OU5cu+qTHLudbA+t36ej3Zj+BTjNzH6G4O6GR4D/B8wn+JYLMEpw\n8k4rpnfCmNKop5Ewpn8H/C93PxyutPl94D/RmXqKIy6Iv65OuLMlte9qOQv4G3d/3d3Hgb8mqJdO\n1FW7MUF69QTw28C3q8qnWU9TxQQp1ZO7vwX8gru/EW4/CJxFC/WUmaTh7kdqLNn7A4I/egi+3Zw7\n+USYORe4+5+Hm8rAXDN71syeM7MPJBWTu/8I+CrwfwiWRF4f3kfkPMLb4br7O0DZzNrqEmwjpi+G\nMaVRT1cT/O52ATPN7CwzexfBiflcgnr6afhasdRTTHFBZ+qq+m6Y7/IT72p59LMTOlCxPda6iiEm\nSK+eCPerPhl36m+vnZgg3Xp6DcDMZhJ8Ufo2LdRTZpJGlcnK/iTwMTP7S4Il1St/CR8Bnqn4/7eB\n1e7+K8CnCZZhTyQmM7uQ4B4hFxJ8k/2EmZ1dVa76cVoxpVZP7v4vBDfnehr47wTJrFaddGJJ6Vbi\n6lhdVdwNc+UUcU4Vf9TtScaUpXqaap8062lS6vVkZhcT3NNosbv/a419GtZTVpMGAO7+Q3f/VXdf\nAPwjwX04Jl1D0BUzua+7+7Ph428DP2dmsZ98pojpQ8B33H08zOYvAjMJ7k54HhwdmOup+EWlElPK\n9YS7b3b32e6+hKApvYeE6qnJuHZ3qq7s2N0wfyX83bwR3mAMjt3V8midTLU9zrpqMaZe4Ecp19Ok\n6vWQ0qynmjGlXU9m1kvQFXWLu38/fL7pespi0jhaiWa2xswWhf+9mWP9dBCcFL9Xse8nzey3w8fv\nAw64e1wLazWKaRfwH82sJ6z4mcA/A38J/Ga4769x4sBY0jHtTqmelgJfN7OTzex5M/uZsCV0EfA3\ndLaeWo3r78zsv8ZdV3bszpbX+LE7W07e1RKO3dXyO8CHzOzMsMvscoL+69jrqo2YZgPf6sRnqomY\nJlWffNOsp5oxZaCeHgVuc/fvVrxE0/WUmQULzew/A48A5xDc5e8gwWDSQ8CpwAvu/ocV++939/Mq\n/n8+8KcEifAk4A/c/W+TismCe5r/clj0f7r7Q2Z2EvAnwMUEt6ntD8ca0owp7Xr6XYK+15PD936h\nE/UUU1ydqKvqu2GWgX6C4z/urpZmdj1Bd1oZeMjdN3XoM9VuTKnVU/jcdwlahzOAvQQz4Z4jpXqq\nE9M/kF49/Xvg7zk2CQSCmYXfoMl6ykzSEBGR7Mti95SIiGSUkoaIiESmpCEiIpEpaYiISGRKGiIi\nEpmShoiIRKakISIikSlpiIhIZP8fRGPhl15L4w8AAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x7f373feda110>"
       ]
      }
     ],
     "prompt_number": 117
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Structured data with Pandas\n",
      "\n",
      "Pandas is both (i) a collection of data structures for structured data and (ii) a bunch of statistical functions.\n",
      "\n",
      "At the center of the Pandas ecosystem is the DataFrame, which is a matrix with indices for rows and columns.\n",
      "For instance, the code below builds DataFrames containing mean monthly temperatures for 3 US cities. (Source: dmi.dk)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pandas as pd\n",
      "\n",
      "daytemps = pd.DataFrame(\n",
      "    index=['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'],\n",
      "    data={  # Note the use of a dictionary here to name the columns\n",
      "        'Baltimore':[6,8,14,19,25,29,31,31,27,21,15,8],\n",
      "        'Los_Angeles':[19,19,19,20,21,22,24,25,25,24,21,19],\n",
      "        'Anchorage':[-6,-3,1,6,12,16,18,17,13,5,-3,-5]}\n",
      "    )\n",
      "nighttemps = pd.DataFrame(\n",
      "    index=['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'],\n",
      "    data={  # Note the use of a dictionary here to name the columns\n",
      "        'Baltimore':[-3,-2,3,8,14,19,22,21,17,10,5,0],\n",
      "        'Los_Angeles':[9,10,10,12,14,15,17,18,17,15,12,9],\n",
      "        'Anchorage':[-13,-11,-8,-2,4,8,11,10,5,-2,-9,-12]}\n",
      "    )\n",
      "print daytemps"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Columns and rows are indexable as below:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print 'Baltimore day temps:'\n",
      "print daytemps['Baltimore'] # Dictionary-style indexing of columns\n",
      "print 'Baltimore day temps:'\n",
      "print daytemps.Baltimore # Columns are available this way if properly named\n",
      "print 'June day temps:'\n",
      "print daytemps.ix['Jun'] # Row-indexing uses the .ix function\n",
      "print 'Summer night temps:'\n",
      "print nighttemps.ix['Jun':'Aug'] # Slicing is also available"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Pandas hooks into matplotlib to plot data. Plots can be done directly from the dataframe. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import matplotlib.pyplot as plt\n",
      "import seaborn as sns\n",
      "import numpy as np\n",
      "%matplotlib inline\n",
      "sns.set_style('ticks')\n",
      "\n",
      "daytemps.plot()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note that Pandas will by default plot to a new figure when asked to plot multiple things (in MATLAB parlance, \"hold\" is off), so we have to do a bit more to get it to plot the day and night temps on the same axes.\n",
      "Keywords for line attributes and labels are also supported"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "daytemps.Baltimore.plot(color='black', label='Day, Baltimore')\n",
      "ax = plt.gca() # Get the current axes\n",
      "daytemps.Los_Angeles.plot(ax=ax, color='red', label='Day, Los Angeles')\n",
      "nighttemps.Baltimore.plot(ax=ax, linestyle='--', color='black', label='Night, Baltimore')\n",
      "nighttemps.Los_Angeles.plot(ax=ax, linestyle='--', color='red', label='Night, Los Angeles')\n",
      "plt.ylabel(u'Temperature [\u00baC]')\n",
      "plt.legend(ncol=2, loc='lower center')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In most respects, DataFrames behave just like numpy arrays, and numpy functions work on them:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print 'Mean day and night temperature differences:'\n",
      "print (daytemps - nighttemps).mean()\n",
      "\n",
      "print 'RMS of day temperatures:'\n",
      "print np.sqrt((daytemps**2).mean(axis=1))\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "__\n",
      "\n",
      "__EX:__ Plot the day temperatures as a horizontal boxplot. (Hint: look at the possible options for plots in _daytemps.plot?_ )\n",
      "\n",
      "Plot a scatterplot of day temperature vs. night temperature in each city. (Hint: A for-loop over the column names might be helpful here...)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Type type type..."
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "-----------\n",
      "\n",
      "Pandas is useful for loading up and analyzing data. Below, we'll load up some data on various cars, contained in a .csv file."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cardata = pd.read_csv('16tstcar.csv')\n",
      "print cardata.ix[0]\n",
      "cardata.plot(kind='scatter', x='Equivalent Test Weight (lbs.)', y='RND_ADJ_FE')\n",
      "\n",
      "cardata.plot(kind='scatter', x='Displacement (L)', y='RND_ADJ_FE')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Our data includes data points for both highway and town cycles, which could screw up our analysis. We should group the data points depending on which cycle they are from. The resulting object from the grouping (carbycycle) acts like a container of dataframes:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "carbycycle = cardata.groupby('Test Category')\n",
      "print carbycycle.RND_ADJ_FE.mean() # Show the mean MPG for each category\n",
      "carhwy = carbycycle.get_group('HWY') # Only use highway data\n",
      "print carhwy.Model"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For this reduced set of data points, we can look for the factors which impact MPG the most by running Ordinary Least Squares:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "olsresult = pd.stats.ols.OLS(\n",
      "    carhwy.RND_ADJ_FE,\n",
      "# Pick out the columns we'd like to analyse\n",
      "    carhwy[['Equivalent Test Weight (lbs.)', \n",
      "        'Displacement (L)', 'Cylinders and Rotors','N/V Ratio',\n",
      "        'Gears', 'Axle Ratio', 'Rated Horsepower']]  \n",
      ")\n",
      "print olsresult.p_value # p-value < 0.05 => Potentially significant"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "__EX:__ Because of the small number of observations, the model above is most likely overfitted. Remove columns from the OLS calculation until you get down to the 3 most significant, then plot MPG versus those values to convince yourself that these are actually significant.\n",
      "\n",
      "For the 3 columns you found before, plot a scatterplot matrix of the data (Hint: pd.scatter_matrix).\n",
      "\n",
      "Do the same factors matter for town driving? (Test Category = FTP)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Typing! Here!"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 123
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Making your own modules\n",
      "\n",
      "To make your own modules, simply place a .py file with the functions or data you want to import in the working directory.\n",
      "I've placed a file called \"myhelpers.py\" in this folder. You can import it as you would any module:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import myhelpers as mh\n",
      "import numpy as np\n",
      "\n",
      "a = np.array([-3, 4, 5, 4, -12, 100, -22, 0])\n",
      "\n",
      "print mh.pos(a)\n",
      "print mh.neg(a)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Below is the code in myhelpers.py"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from numpy import where\n",
      "\n",
      "\n",
      "def pos(x):\n",
      "    \"\"\"\n",
      "        Returns (x)_+, i.e.\n",
      "        pos(x) = x if x > 0, else 0\n",
      "\n",
      "        Input:\n",
      "            x: numeric or array_like.\n",
      "\n",
      "        Output:\n",
      "            y: arraylike copy of x with negative entries set to 0\n",
      "    \"\"\"\n",
      "    return where(x > 0, x, 0)\n",
      "\n",
      "\n",
      "def neg(x):\n",
      "    \"\"\"\n",
      "        Returns (x)_-, i.e.\n",
      "        neg(x) = x if x < 0, else 0\n",
      "\n",
      "        Input:\n",
      "            x: numeric or array_like.\n",
      "\n",
      "        Output:\n",
      "            y: arraylike copy of x with positive entries set to 0\n",
      "    \"\"\"\n",
      "    return where(x < 0, x, 0)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}