HDF5 with pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# About Pandas\n",
    "\n",
    "http://pandas.pydata.org/\n",
    "\n",
    "Open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.\n",
    "\n",
    "Excerpt of Pandas' features:\n",
    "\n",
    "- A set of labeled array data structures, the primary of which are Series and DataFrame\n",
    "- Index objects enabling both simple axis indexing and multi-level / hierarchical axis indexing\n",
    "- An integrated group by engine for aggregating and transforming data sets\n",
    "- **Input/Output tools: loading tabular data from flat files (CSV, delimited, Excel 2003), and saving and loading pandas objects from the fast and efficient PyTables/HDF5 format.**\n",
    "- ...\n",
    "\n",
    "So internally, *pandas* uses *PyTables* for storing data in HDF5.\n",
    "\n",
    "http://pandas.pydata.org/pandas-docs/stable/io.html#hdf5-pytables\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading some example data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas_datareader.data as web\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "start = datetime.datetime(2010, 1, 1)\n",
    "end = datetime.datetime(2013, 1, 27)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = web.DataReader(\"F\", 'yahoo', start, end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2010-01-04</th>\n",
       "      <td>10.17</td>\n",
       "      <td>10.28</td>\n",
       "      <td>10.05</td>\n",
       "      <td>10.28</td>\n",
       "      <td>60855800</td>\n",
       "      <td>8.418735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-05</th>\n",
       "      <td>10.45</td>\n",
       "      <td>11.24</td>\n",
       "      <td>10.40</td>\n",
       "      <td>10.96</td>\n",
       "      <td>215620200</td>\n",
       "      <td>8.975616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-06</th>\n",
       "      <td>11.21</td>\n",
       "      <td>11.46</td>\n",
       "      <td>11.13</td>\n",
       "      <td>11.37</td>\n",
       "      <td>200070600</td>\n",
       "      <td>9.311383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-07</th>\n",
       "      <td>11.46</td>\n",
       "      <td>11.69</td>\n",
       "      <td>11.32</td>\n",
       "      <td>11.66</td>\n",
       "      <td>130201700</td>\n",
       "      <td>9.548876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-08</th>\n",
       "      <td>11.67</td>\n",
       "      <td>11.74</td>\n",
       "      <td>11.46</td>\n",
       "      <td>11.69</td>\n",
       "      <td>130463000</td>\n",
       "      <td>9.573444</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Open   High    Low  Close     Volume  Adj Close\n",
       "Date                                                        \n",
       "2010-01-04  10.17  10.28  10.05  10.28   60855800   8.418735\n",
       "2010-01-05  10.45  11.24  10.40  10.96  215620200   8.975616\n",
       "2010-01-06  11.21  11.46  11.13  11.37  200070600   9.311383\n",
       "2010-01-07  11.46  11.69  11.32  11.66  130201700   9.548876\n",
       "2010-01-08  11.67  11.74  11.46  11.69  130463000   9.573444"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store = pd.HDFStore('example-pandas.h5', 'a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store['yahoo'] = df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Natural naming is supported:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2010-01-04</th>\n",
       "      <td>10.17</td>\n",
       "      <td>10.28</td>\n",
       "      <td>10.05</td>\n",
       "      <td>10.28</td>\n",
       "      <td>60855800</td>\n",
       "      <td>8.418735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-05</th>\n",
       "      <td>10.45</td>\n",
       "      <td>11.24</td>\n",
       "      <td>10.40</td>\n",
       "      <td>10.96</td>\n",
       "      <td>215620200</td>\n",
       "      <td>8.975616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-06</th>\n",
       "      <td>11.21</td>\n",
       "      <td>11.46</td>\n",
       "      <td>11.13</td>\n",
       "      <td>11.37</td>\n",
       "      <td>200070600</td>\n",
       "      <td>9.311383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-07</th>\n",
       "      <td>11.46</td>\n",
       "      <td>11.69</td>\n",
       "      <td>11.32</td>\n",
       "      <td>11.66</td>\n",
       "      <td>130201700</td>\n",
       "      <td>9.548876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-08</th>\n",
       "      <td>11.67</td>\n",
       "      <td>11.74</td>\n",
       "      <td>11.46</td>\n",
       "      <td>11.69</td>\n",
       "      <td>130463000</td>\n",
       "      <td>9.573444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-11</th>\n",
       "      <td>11.90</td>\n",
       "      <td>12.14</td>\n",
       "      <td>11.78</td>\n",
       "      <td>12.11</td>\n",
       "      <td>170626200</td>\n",
       "      <td>9.917400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-12</th>\n",
       "      <td>11.98</td>\n",
       "      <td>12.03</td>\n",
       "      <td>11.72</td>\n",
       "      <td>11.87</td>\n",
       "      <td>162995900</td>\n",
       "      <td>9.720854</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-13</th>\n",
       "      <td>11.91</td>\n",
       "      <td>11.93</td>\n",
       "      <td>11.47</td>\n",
       "      <td>11.68</td>\n",
       "      <td>154527100</td>\n",
       "      <td>9.565255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-14</th>\n",
       "      <td>11.65</td>\n",
       "      <td>11.86</td>\n",
       "      <td>11.51</td>\n",
       "      <td>11.76</td>\n",
       "      <td>116531200</td>\n",
       "      <td>9.630771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-15</th>\n",
       "      <td>11.74</td>\n",
       "      <td>11.76</td>\n",
       "      <td>11.55</td>\n",
       "      <td>11.60</td>\n",
       "      <td>96149800</td>\n",
       "      <td>9.499740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-19</th>\n",
       "      <td>11.51</td>\n",
       "      <td>11.83</td>\n",
       "      <td>11.46</td>\n",
       "      <td>11.75</td>\n",
       "      <td>65934000</td>\n",
       "      <td>9.622581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-20</th>\n",
       "      <td>11.68</td>\n",
       "      <td>11.69</td>\n",
       "      <td>11.50</td>\n",
       "      <td>11.51</td>\n",
       "      <td>71649500</td>\n",
       "      <td>9.426035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-21</th>\n",
       "      <td>11.53</td>\n",
       "      <td>11.62</td>\n",
       "      <td>11.01</td>\n",
       "      <td>11.18</td>\n",
       "      <td>121451400</td>\n",
       "      <td>9.155784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-22</th>\n",
       "      <td>11.01</td>\n",
       "      <td>11.12</td>\n",
       "      <td>10.41</td>\n",
       "      <td>10.52</td>\n",
       "      <td>161530100</td>\n",
       "      <td>8.615282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-25</th>\n",
       "      <td>10.73</td>\n",
       "      <td>11.10</td>\n",
       "      <td>10.61</td>\n",
       "      <td>11.03</td>\n",
       "      <td>121621500</td>\n",
       "      <td>9.032942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-26</th>\n",
       "      <td>11.17</td>\n",
       "      <td>11.46</td>\n",
       "      <td>11.07</td>\n",
       "      <td>11.19</td>\n",
       "      <td>108250500</td>\n",
       "      <td>9.163973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-27</th>\n",
       "      <td>11.57</td>\n",
       "      <td>11.62</td>\n",
       "      <td>11.22</td>\n",
       "      <td>11.55</td>\n",
       "      <td>105091600</td>\n",
       "      <td>9.458793</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-28</th>\n",
       "      <td>11.90</td>\n",
       "      <td>11.95</td>\n",
       "      <td>11.27</td>\n",
       "      <td>11.41</td>\n",
       "      <td>203320000</td>\n",
       "      <td>9.344140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-29</th>\n",
       "      <td>11.60</td>\n",
       "      <td>11.61</td>\n",
       "      <td>10.70</td>\n",
       "      <td>10.84</td>\n",
       "      <td>159741200</td>\n",
       "      <td>8.877343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-01</th>\n",
       "      <td>11.14</td>\n",
       "      <td>11.18</td>\n",
       "      <td>10.93</td>\n",
       "      <td>11.12</td>\n",
       "      <td>82748200</td>\n",
       "      <td>9.106647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-02</th>\n",
       "      <td>11.26</td>\n",
       "      <td>11.52</td>\n",
       "      <td>11.19</td>\n",
       "      <td>11.39</td>\n",
       "      <td>119714900</td>\n",
       "      <td>9.327762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-03</th>\n",
       "      <td>11.49</td>\n",
       "      <td>11.66</td>\n",
       "      <td>11.42</td>\n",
       "      <td>11.64</td>\n",
       "      <td>90125500</td>\n",
       "      <td>9.532498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-04</th>\n",
       "      <td>11.49</td>\n",
       "      <td>11.53</td>\n",
       "      <td>11.00</td>\n",
       "      <td>11.06</td>\n",
       "      <td>129792200</td>\n",
       "      <td>9.057511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-05</th>\n",
       "      <td>10.97</td>\n",
       "      <td>11.11</td>\n",
       "      <td>10.49</td>\n",
       "      <td>10.91</td>\n",
       "      <td>181535200</td>\n",
       "      <td>8.934669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-08</th>\n",
       "      <td>11.09</td>\n",
       "      <td>11.32</td>\n",
       "      <td>10.88</td>\n",
       "      <td>10.97</td>\n",
       "      <td>92031400</td>\n",
       "      <td>8.983806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-09</th>\n",
       "      <td>11.18</td>\n",
       "      <td>11.22</td>\n",
       "      <td>11.02</td>\n",
       "      <td>11.15</td>\n",
       "      <td>83207100</td>\n",
       "      <td>9.131215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-10</th>\n",
       "      <td>11.12</td>\n",
       "      <td>11.14</td>\n",
       "      <td>10.90</td>\n",
       "      <td>10.94</td>\n",
       "      <td>73395600</td>\n",
       "      <td>8.959237</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-11</th>\n",
       "      <td>11.00</td>\n",
       "      <td>11.19</td>\n",
       "      <td>10.88</td>\n",
       "      <td>11.18</td>\n",
       "      <td>65116200</td>\n",
       "      <td>9.155784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-12</th>\n",
       "      <td>10.92</td>\n",
       "      <td>11.18</td>\n",
       "      <td>10.85</td>\n",
       "      <td>11.12</td>\n",
       "      <td>69465400</td>\n",
       "      <td>9.106647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-16</th>\n",
       "      <td>11.21</td>\n",
       "      <td>11.38</td>\n",
       "      <td>11.11</td>\n",
       "      <td>11.32</td>\n",
       "      <td>62537500</td>\n",
       "      <td>9.270435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-12</th>\n",
       "      <td>11.52</td>\n",
       "      <td>11.56</td>\n",
       "      <td>11.43</td>\n",
       "      <td>11.47</td>\n",
       "      <td>31099900</td>\n",
       "      <td>9.568676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-13</th>\n",
       "      <td>11.46</td>\n",
       "      <td>11.50</td>\n",
       "      <td>11.21</td>\n",
       "      <td>11.27</td>\n",
       "      <td>35443200</td>\n",
       "      <td>9.401829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-14</th>\n",
       "      <td>11.27</td>\n",
       "      <td>11.27</td>\n",
       "      <td>11.03</td>\n",
       "      <td>11.10</td>\n",
       "      <td>36933500</td>\n",
       "      <td>9.260009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-17</th>\n",
       "      <td>11.16</td>\n",
       "      <td>11.41</td>\n",
       "      <td>11.14</td>\n",
       "      <td>11.39</td>\n",
       "      <td>46983300</td>\n",
       "      <td>9.501937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-18</th>\n",
       "      <td>11.48</td>\n",
       "      <td>11.68</td>\n",
       "      <td>11.40</td>\n",
       "      <td>11.67</td>\n",
       "      <td>61810400</td>\n",
       "      <td>9.735523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-19</th>\n",
       "      <td>11.79</td>\n",
       "      <td>11.85</td>\n",
       "      <td>11.62</td>\n",
       "      <td>11.73</td>\n",
       "      <td>54884700</td>\n",
       "      <td>9.785576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-20</th>\n",
       "      <td>11.74</td>\n",
       "      <td>11.80</td>\n",
       "      <td>11.58</td>\n",
       "      <td>11.77</td>\n",
       "      <td>47750100</td>\n",
       "      <td>9.818946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-21</th>\n",
       "      <td>11.55</td>\n",
       "      <td>11.86</td>\n",
       "      <td>11.47</td>\n",
       "      <td>11.86</td>\n",
       "      <td>94489300</td>\n",
       "      <td>9.894027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-24</th>\n",
       "      <td>11.67</td>\n",
       "      <td>12.40</td>\n",
       "      <td>11.67</td>\n",
       "      <td>12.40</td>\n",
       "      <td>91734900</td>\n",
       "      <td>10.344514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-26</th>\n",
       "      <td>12.31</td>\n",
       "      <td>12.79</td>\n",
       "      <td>12.31</td>\n",
       "      <td>12.79</td>\n",
       "      <td>140331900</td>\n",
       "      <td>10.669866</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-27</th>\n",
       "      <td>12.79</td>\n",
       "      <td>12.81</td>\n",
       "      <td>12.36</td>\n",
       "      <td>12.76</td>\n",
       "      <td>108315100</td>\n",
       "      <td>10.644839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-28</th>\n",
       "      <td>12.55</td>\n",
       "      <td>12.88</td>\n",
       "      <td>12.52</td>\n",
       "      <td>12.87</td>\n",
       "      <td>95668600</td>\n",
       "      <td>10.736605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012-12-31</th>\n",
       "      <td>12.88</td>\n",
       "      <td>13.08</td>\n",
       "      <td>12.76</td>\n",
       "      <td>12.95</td>\n",
       "      <td>106908900</td>\n",
       "      <td>10.803343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02</th>\n",
       "      <td>13.23</td>\n",
       "      <td>13.28</td>\n",
       "      <td>13.00</td>\n",
       "      <td>13.20</td>\n",
       "      <td>75274700</td>\n",
       "      <td>11.011902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-03</th>\n",
       "      <td>13.24</td>\n",
       "      <td>13.70</td>\n",
       "      <td>13.05</td>\n",
       "      <td>13.46</td>\n",
       "      <td>121284700</td>\n",
       "      <td>11.228803</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-04</th>\n",
       "      <td>13.51</td>\n",
       "      <td>13.61</td>\n",
       "      <td>13.35</td>\n",
       "      <td>13.57</td>\n",
       "      <td>54669900</td>\n",
       "      <td>11.320569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-07</th>\n",
       "      <td>13.52</td>\n",
       "      <td>13.58</td>\n",
       "      <td>13.35</td>\n",
       "      <td>13.43</td>\n",
       "      <td>43482400</td>\n",
       "      <td>11.203776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-08</th>\n",
       "      <td>13.38</td>\n",
       "      <td>13.43</td>\n",
       "      <td>13.20</td>\n",
       "      <td>13.35</td>\n",
       "      <td>46336200</td>\n",
       "      <td>11.137038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-09</th>\n",
       "      <td>13.40</td>\n",
       "      <td>13.60</td>\n",
       "      <td>13.39</td>\n",
       "      <td>13.47</td>\n",
       "      <td>36973900</td>\n",
       "      <td>11.237146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-10</th>\n",
       "      <td>13.78</td>\n",
       "      <td>13.94</td>\n",
       "      <td>13.68</td>\n",
       "      <td>13.83</td>\n",
       "      <td>84884300</td>\n",
       "      <td>11.537470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-11</th>\n",
       "      <td>14.00</td>\n",
       "      <td>14.07</td>\n",
       "      <td>13.90</td>\n",
       "      <td>14.00</td>\n",
       "      <td>67641200</td>\n",
       "      <td>11.679290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-14</th>\n",
       "      <td>14.04</td>\n",
       "      <td>14.08</td>\n",
       "      <td>13.86</td>\n",
       "      <td>13.99</td>\n",
       "      <td>46224800</td>\n",
       "      <td>11.670948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-15</th>\n",
       "      <td>13.91</td>\n",
       "      <td>14.30</td>\n",
       "      <td>13.89</td>\n",
       "      <td>14.30</td>\n",
       "      <td>55425000</td>\n",
       "      <td>11.929561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-16</th>\n",
       "      <td>14.16</td>\n",
       "      <td>14.25</td>\n",
       "      <td>14.01</td>\n",
       "      <td>14.22</td>\n",
       "      <td>51453400</td>\n",
       "      <td>11.862822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-17</th>\n",
       "      <td>14.27</td>\n",
       "      <td>14.30</td>\n",
       "      <td>14.07</td>\n",
       "      <td>14.22</td>\n",
       "      <td>44299300</td>\n",
       "      <td>11.862822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-18</th>\n",
       "      <td>14.10</td>\n",
       "      <td>14.13</td>\n",
       "      <td>13.97</td>\n",
       "      <td>14.11</td>\n",
       "      <td>45716100</td>\n",
       "      <td>11.771056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-22</th>\n",
       "      <td>14.06</td>\n",
       "      <td>14.19</td>\n",
       "      <td>14.00</td>\n",
       "      <td>14.17</td>\n",
       "      <td>35467900</td>\n",
       "      <td>11.821110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-23</th>\n",
       "      <td>14.00</td>\n",
       "      <td>14.02</td>\n",
       "      <td>13.79</td>\n",
       "      <td>13.88</td>\n",
       "      <td>58122200</td>\n",
       "      <td>11.579182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-24</th>\n",
       "      <td>13.82</td>\n",
       "      <td>13.98</td>\n",
       "      <td>13.81</td>\n",
       "      <td>13.87</td>\n",
       "      <td>42532700</td>\n",
       "      <td>11.570840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-25</th>\n",
       "      <td>13.83</td>\n",
       "      <td>13.84</td>\n",
       "      <td>13.64</td>\n",
       "      <td>13.68</td>\n",
       "      <td>53405400</td>\n",
       "      <td>11.412335</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>771 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             Open   High    Low  Close     Volume  Adj Close\n",
       "Date                                                        \n",
       "2010-01-04  10.17  10.28  10.05  10.28   60855800   8.418735\n",
       "2010-01-05  10.45  11.24  10.40  10.96  215620200   8.975616\n",
       "2010-01-06  11.21  11.46  11.13  11.37  200070600   9.311383\n",
       "2010-01-07  11.46  11.69  11.32  11.66  130201700   9.548876\n",
       "2010-01-08  11.67  11.74  11.46  11.69  130463000   9.573444\n",
       "2010-01-11  11.90  12.14  11.78  12.11  170626200   9.917400\n",
       "2010-01-12  11.98  12.03  11.72  11.87  162995900   9.720854\n",
       "2010-01-13  11.91  11.93  11.47  11.68  154527100   9.565255\n",
       "2010-01-14  11.65  11.86  11.51  11.76  116531200   9.630771\n",
       "2010-01-15  11.74  11.76  11.55  11.60   96149800   9.499740\n",
       "2010-01-19  11.51  11.83  11.46  11.75   65934000   9.622581\n",
       "2010-01-20  11.68  11.69  11.50  11.51   71649500   9.426035\n",
       "2010-01-21  11.53  11.62  11.01  11.18  121451400   9.155784\n",
       "2010-01-22  11.01  11.12  10.41  10.52  161530100   8.615282\n",
       "2010-01-25  10.73  11.10  10.61  11.03  121621500   9.032942\n",
       "2010-01-26  11.17  11.46  11.07  11.19  108250500   9.163973\n",
       "2010-01-27  11.57  11.62  11.22  11.55  105091600   9.458793\n",
       "2010-01-28  11.90  11.95  11.27  11.41  203320000   9.344140\n",
       "2010-01-29  11.60  11.61  10.70  10.84  159741200   8.877343\n",
       "2010-02-01  11.14  11.18  10.93  11.12   82748200   9.106647\n",
       "2010-02-02  11.26  11.52  11.19  11.39  119714900   9.327762\n",
       "2010-02-03  11.49  11.66  11.42  11.64   90125500   9.532498\n",
       "2010-02-04  11.49  11.53  11.00  11.06  129792200   9.057511\n",
       "2010-02-05  10.97  11.11  10.49  10.91  181535200   8.934669\n",
       "2010-02-08  11.09  11.32  10.88  10.97   92031400   8.983806\n",
       "2010-02-09  11.18  11.22  11.02  11.15   83207100   9.131215\n",
       "2010-02-10  11.12  11.14  10.90  10.94   73395600   8.959237\n",
       "2010-02-11  11.00  11.19  10.88  11.18   65116200   9.155784\n",
       "2010-02-12  10.92  11.18  10.85  11.12   69465400   9.106647\n",
       "2010-02-16  11.21  11.38  11.11  11.32   62537500   9.270435\n",
       "...           ...    ...    ...    ...        ...        ...\n",
       "2012-12-12  11.52  11.56  11.43  11.47   31099900   9.568676\n",
       "2012-12-13  11.46  11.50  11.21  11.27   35443200   9.401829\n",
       "2012-12-14  11.27  11.27  11.03  11.10   36933500   9.260009\n",
       "2012-12-17  11.16  11.41  11.14  11.39   46983300   9.501937\n",
       "2012-12-18  11.48  11.68  11.40  11.67   61810400   9.735523\n",
       "2012-12-19  11.79  11.85  11.62  11.73   54884700   9.785576\n",
       "2012-12-20  11.74  11.80  11.58  11.77   47750100   9.818946\n",
       "2012-12-21  11.55  11.86  11.47  11.86   94489300   9.894027\n",
       "2012-12-24  11.67  12.40  11.67  12.40   91734900  10.344514\n",
       "2012-12-26  12.31  12.79  12.31  12.79  140331900  10.669866\n",
       "2012-12-27  12.79  12.81  12.36  12.76  108315100  10.644839\n",
       "2012-12-28  12.55  12.88  12.52  12.87   95668600  10.736605\n",
       "2012-12-31  12.88  13.08  12.76  12.95  106908900  10.803343\n",
       "2013-01-02  13.23  13.28  13.00  13.20   75274700  11.011902\n",
       "2013-01-03  13.24  13.70  13.05  13.46  121284700  11.228803\n",
       "2013-01-04  13.51  13.61  13.35  13.57   54669900  11.320569\n",
       "2013-01-07  13.52  13.58  13.35  13.43   43482400  11.203776\n",
       "2013-01-08  13.38  13.43  13.20  13.35   46336200  11.137038\n",
       "2013-01-09  13.40  13.60  13.39  13.47   36973900  11.237146\n",
       "2013-01-10  13.78  13.94  13.68  13.83   84884300  11.537470\n",
       "2013-01-11  14.00  14.07  13.90  14.00   67641200  11.679290\n",
       "2013-01-14  14.04  14.08  13.86  13.99   46224800  11.670948\n",
       "2013-01-15  13.91  14.30  13.89  14.30   55425000  11.929561\n",
       "2013-01-16  14.16  14.25  14.01  14.22   51453400  11.862822\n",
       "2013-01-17  14.27  14.30  14.07  14.22   44299300  11.862822\n",
       "2013-01-18  14.10  14.13  13.97  14.11   45716100  11.771056\n",
       "2013-01-22  14.06  14.19  14.00  14.17   35467900  11.821110\n",
       "2013-01-23  14.00  14.02  13.79  13.88   58122200  11.579182\n",
       "2013-01-24  13.82  13.98  13.81  13.87   42532700  11.570840\n",
       "2013-01-25  13.83  13.84  13.64  13.68   53405400  11.412335\n",
       "\n",
       "[771 rows x 6 columns]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.yahoo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access to more PyTable-like view:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "/yahoo (Group) ''\n",
       "  children := ['axis0' (Array), 'axis1' (Array), 'block0_values' (Array), 'block0_items' (Array), 'block1_values' (Array), 'block1_items' (Array)]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.root.yahoo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "/yahoo._v_attrs (AttributeSet), 12 attributes:\n",
       "   [CLASS := 'GROUP',\n",
       "    TITLE := '',\n",
       "    VERSION := '1.0',\n",
       "    axis0_variety := 'regular',\n",
       "    axis1_variety := 'regular',\n",
       "    block0_items_variety := 'regular',\n",
       "    block1_items_variety := 'regular',\n",
       "    encoding := 'UTF-8',\n",
       "    nblocks := 2,\n",
       "    ndim := 2,\n",
       "    pandas_type := 'frame',\n",
       "    pandas_version := '0.15.2']"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.root.yahoo._v_attrs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas introduces own attributes in order to provide interface to `pandas.Series`, `pandas.DataFrame`, .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See HDFView, it's a customized format on top of HDF5.\n",
    "This format also supports indexed search.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shortcut for storing data from a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.to_hdf(\"example-pandas-shortcut.h5\", 'table', append=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shortcut for reading:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_hdf(\"example-pandas-shortcut.h5\", \"table\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Indexed search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store = pd.HDFStore('example-pandas.h5', 'a')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No index yet! And it's in \"fixed format\", so no queries possible:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "cannot pass a where specification when reading from a Fixed format store. this store must be selected in its entirety",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-46-517779c12539>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mstore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mselect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"yahoo\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"Open>10 & Open<11\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/home/mcrot/miniconda3/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py\u001b[0m in \u001b[0;36mselect\u001b[0;34m(self, key, where, start, stop, columns, iterator, chunksize, auto_close, **kwargs)\u001b[0m\n\u001b[1;32m    722\u001b[0m                            chunksize=chunksize, auto_close=auto_close)\n\u001b[1;32m    723\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 724\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mit\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    725\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    726\u001b[0m     def select_as_coordinates(\n",
      "\u001b[0;32m/home/mcrot/miniconda3/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py\u001b[0m in \u001b[0;36mget_result\u001b[0;34m(self, coordinates)\u001b[0m\n\u001b[1;32m   1421\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1422\u001b[0m         \u001b[0;31m# directly return the result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1423\u001b[0;31m         \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstop\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwhere\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1424\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1425\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresults\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mcrot/miniconda3/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py\u001b[0m in \u001b[0;36mfunc\u001b[0;34m(_start, _stop, _where)\u001b[0m\n\u001b[1;32m    715\u001b[0m             return s.read(start=_start, stop=_stop,\n\u001b[1;32m    716\u001b[0m                           \u001b[0mwhere\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0m_where\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 717\u001b[0;31m                           columns=columns, **kwargs)\n\u001b[0m\u001b[1;32m    718\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    719\u001b[0m         \u001b[0;31m# create the iterator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mcrot/miniconda3/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, start, stop, **kwargs)\u001b[0m\n\u001b[1;32m   2833\u001b[0m         \u001b[0;31m# start, stop applied to rows, so 0th axis only\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2834\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2835\u001b[0;31m         \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalidate_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2836\u001b[0m         \u001b[0mselect_axis\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mobj_type\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_block_manager_axis\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2837\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mcrot/miniconda3/envs/py36/lib/python3.6/site-packages/pandas/io/pytables.py\u001b[0m in \u001b[0;36mvalidate_read\u001b[0;34m(self, kwargs)\u001b[0m\n\u001b[1;32m   2372\u001b[0m         \u001b[0mwhere\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'where'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2373\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mwhere\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2374\u001b[0;31m             raise TypeError(\"cannot pass a where specification when reading \"\n\u001b[0m\u001b[1;32m   2375\u001b[0m                             \u001b[0;34m\"from a Fixed format store. this store must be \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2376\u001b[0m                             \"selected in its entirety\")\n",
      "\u001b[0;31mTypeError\u001b[0m: cannot pass a where specification when reading from a Fixed format store. this store must be selected in its entirety"
     ]
    }
   ],
   "source": [
    "store.select(\"yahoo\", \"Open>10 & Open<11\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store.append('yahoo_tableformat', df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show in HDFView, different format!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store.append('yahoo_indexed', df, data_columns=['Open', 'Close'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store.flush()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2010-01-04</th>\n",
       "      <td>10.17</td>\n",
       "      <td>10.28</td>\n",
       "      <td>10.05</td>\n",
       "      <td>10.28</td>\n",
       "      <td>60855800</td>\n",
       "      <td>8.418735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-05</th>\n",
       "      <td>10.45</td>\n",
       "      <td>11.24</td>\n",
       "      <td>10.40</td>\n",
       "      <td>10.96</td>\n",
       "      <td>215620200</td>\n",
       "      <td>8.975616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-25</th>\n",
       "      <td>10.73</td>\n",
       "      <td>11.10</td>\n",
       "      <td>10.61</td>\n",
       "      <td>11.03</td>\n",
       "      <td>121621500</td>\n",
       "      <td>9.032942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-05</th>\n",
       "      <td>10.97</td>\n",
       "      <td>11.11</td>\n",
       "      <td>10.49</td>\n",
       "      <td>10.91</td>\n",
       "      <td>181535200</td>\n",
       "      <td>8.934669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-12</th>\n",
       "      <td>10.92</td>\n",
       "      <td>11.18</td>\n",
       "      <td>10.85</td>\n",
       "      <td>11.12</td>\n",
       "      <td>69465400</td>\n",
       "      <td>9.106647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-05-21</th>\n",
       "      <td>10.25</td>\n",
       "      <td>11.30</td>\n",
       "      <td>10.17</td>\n",
       "      <td>11.26</td>\n",
       "      <td>174455100</td>\n",
       "      <td>9.221299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-05-25</th>\n",
       "      <td>10.47</td>\n",
       "      <td>11.05</td>\n",
       "      <td>10.42</td>\n",
       "      <td>11.02</td>\n",
       "      <td>137858600</td>\n",
       "      <td>9.024753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-24</th>\n",
       "      <td>10.99</td>\n",
       "      <td>11.03</td>\n",
       "      <td>10.64</td>\n",
       "      <td>10.78</td>\n",
       "      <td>74449200</td>\n",
       "      <td>8.828206</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-25</th>\n",
       "      <td>10.75</td>\n",
       "      <td>10.77</td>\n",
       "      <td>10.42</td>\n",
       "      <td>10.75</td>\n",
       "      <td>148168400</td>\n",
       "      <td>8.803638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-28</th>\n",
       "      <td>10.72</td>\n",
       "      <td>10.76</td>\n",
       "      <td>10.43</td>\n",
       "      <td>10.43</td>\n",
       "      <td>57994700</td>\n",
       "      <td>8.541577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-29</th>\n",
       "      <td>10.12</td>\n",
       "      <td>10.14</td>\n",
       "      <td>9.75</td>\n",
       "      <td>9.88</td>\n",
       "      <td>124866100</td>\n",
       "      <td>8.091158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-30</th>\n",
       "      <td>10.10</td>\n",
       "      <td>10.52</td>\n",
       "      <td>10.04</td>\n",
       "      <td>10.08</td>\n",
       "      <td>182831300</td>\n",
       "      <td>8.254946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-01</th>\n",
       "      <td>10.23</td>\n",
       "      <td>10.63</td>\n",
       "      <td>10.02</td>\n",
       "      <td>10.57</td>\n",
       "      <td>153383000</td>\n",
       "      <td>8.656228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-02</th>\n",
       "      <td>10.58</td>\n",
       "      <td>10.63</td>\n",
       "      <td>10.15</td>\n",
       "      <td>10.28</td>\n",
       "      <td>71607100</td>\n",
       "      <td>8.418735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-06</th>\n",
       "      <td>10.42</td>\n",
       "      <td>10.57</td>\n",
       "      <td>10.04</td>\n",
       "      <td>10.16</td>\n",
       "      <td>81246900</td>\n",
       "      <td>8.320462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-07</th>\n",
       "      <td>10.21</td>\n",
       "      <td>10.63</td>\n",
       "      <td>10.17</td>\n",
       "      <td>10.59</td>\n",
       "      <td>72565300</td>\n",
       "      <td>8.672607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-08</th>\n",
       "      <td>10.72</td>\n",
       "      <td>10.83</td>\n",
       "      <td>10.45</td>\n",
       "      <td>10.62</td>\n",
       "      <td>62355500</td>\n",
       "      <td>8.697175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-09</th>\n",
       "      <td>10.58</td>\n",
       "      <td>10.94</td>\n",
       "      <td>10.54</td>\n",
       "      <td>10.85</td>\n",
       "      <td>50732700</td>\n",
       "      <td>8.885533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-12</th>\n",
       "      <td>10.84</td>\n",
       "      <td>11.13</td>\n",
       "      <td>10.83</td>\n",
       "      <td>11.10</td>\n",
       "      <td>56354100</td>\n",
       "      <td>9.090269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-04</th>\n",
       "      <td>10.17</td>\n",
       "      <td>10.28</td>\n",
       "      <td>10.05</td>\n",
       "      <td>10.28</td>\n",
       "      <td>60855800</td>\n",
       "      <td>8.418735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-05</th>\n",
       "      <td>10.45</td>\n",
       "      <td>11.24</td>\n",
       "      <td>10.40</td>\n",
       "      <td>10.96</td>\n",
       "      <td>215620200</td>\n",
       "      <td>8.975616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-25</th>\n",
       "      <td>10.73</td>\n",
       "      <td>11.10</td>\n",
       "      <td>10.61</td>\n",
       "      <td>11.03</td>\n",
       "      <td>121621500</td>\n",
       "      <td>9.032942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-05</th>\n",
       "      <td>10.97</td>\n",
       "      <td>11.11</td>\n",
       "      <td>10.49</td>\n",
       "      <td>10.91</td>\n",
       "      <td>181535200</td>\n",
       "      <td>8.934669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-12</th>\n",
       "      <td>10.92</td>\n",
       "      <td>11.18</td>\n",
       "      <td>10.85</td>\n",
       "      <td>11.12</td>\n",
       "      <td>69465400</td>\n",
       "      <td>9.106647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-05-21</th>\n",
       "      <td>10.25</td>\n",
       "      <td>11.30</td>\n",
       "      <td>10.17</td>\n",
       "      <td>11.26</td>\n",
       "      <td>174455100</td>\n",
       "      <td>9.221299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-05-25</th>\n",
       "      <td>10.47</td>\n",
       "      <td>11.05</td>\n",
       "      <td>10.42</td>\n",
       "      <td>11.02</td>\n",
       "      <td>137858600</td>\n",
       "      <td>9.024753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-24</th>\n",
       "      <td>10.99</td>\n",
       "      <td>11.03</td>\n",
       "      <td>10.64</td>\n",
       "      <td>10.78</td>\n",
       "      <td>74449200</td>\n",
       "      <td>8.828206</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-25</th>\n",
       "      <td>10.75</td>\n",
       "      <td>10.77</td>\n",
       "      <td>10.42</td>\n",
       "      <td>10.75</td>\n",
       "      <td>148168400</td>\n",
       "      <td>8.803638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-28</th>\n",
       "      <td>10.72</td>\n",
       "      <td>10.76</td>\n",
       "      <td>10.43</td>\n",
       "      <td>10.43</td>\n",
       "      <td>57994700</td>\n",
       "      <td>8.541577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-29</th>\n",
       "      <td>10.12</td>\n",
       "      <td>10.14</td>\n",
       "      <td>9.75</td>\n",
       "      <td>9.88</td>\n",
       "      <td>124866100</td>\n",
       "      <td>8.091158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-30</th>\n",
       "      <td>10.10</td>\n",
       "      <td>10.52</td>\n",
       "      <td>10.04</td>\n",
       "      <td>10.08</td>\n",
       "      <td>182831300</td>\n",
       "      <td>8.254946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-01</th>\n",
       "      <td>10.23</td>\n",
       "      <td>10.63</td>\n",
       "      <td>10.02</td>\n",
       "      <td>10.57</td>\n",
       "      <td>153383000</td>\n",
       "      <td>8.656228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-02</th>\n",
       "      <td>10.58</td>\n",
       "      <td>10.63</td>\n",
       "      <td>10.15</td>\n",
       "      <td>10.28</td>\n",
       "      <td>71607100</td>\n",
       "      <td>8.418735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-06</th>\n",
       "      <td>10.42</td>\n",
       "      <td>10.57</td>\n",
       "      <td>10.04</td>\n",
       "      <td>10.16</td>\n",
       "      <td>81246900</td>\n",
       "      <td>8.320462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-07</th>\n",
       "      <td>10.21</td>\n",
       "      <td>10.63</td>\n",
       "      <td>10.17</td>\n",
       "      <td>10.59</td>\n",
       "      <td>72565300</td>\n",
       "      <td>8.672607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-08</th>\n",
       "      <td>10.72</td>\n",
       "      <td>10.83</td>\n",
       "      <td>10.45</td>\n",
       "      <td>10.62</td>\n",
       "      <td>62355500</td>\n",
       "      <td>8.697175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-09</th>\n",
       "      <td>10.58</td>\n",
       "      <td>10.94</td>\n",
       "      <td>10.54</td>\n",
       "      <td>10.85</td>\n",
       "      <td>50732700</td>\n",
       "      <td>8.885533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-12</th>\n",
       "      <td>10.84</td>\n",
       "      <td>11.13</td>\n",
       "      <td>10.83</td>\n",
       "      <td>11.10</td>\n",
       "      <td>56354100</td>\n",
       "      <td>9.090269</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Open   High    Low  Close     Volume  Adj Close\n",
       "Date                                                        \n",
       "2010-01-04  10.17  10.28  10.05  10.28   60855800   8.418735\n",
       "2010-01-05  10.45  11.24  10.40  10.96  215620200   8.975616\n",
       "2010-01-25  10.73  11.10  10.61  11.03  121621500   9.032942\n",
       "2010-02-05  10.97  11.11  10.49  10.91  181535200   8.934669\n",
       "2010-02-12  10.92  11.18  10.85  11.12   69465400   9.106647\n",
       "2010-05-21  10.25  11.30  10.17  11.26  174455100   9.221299\n",
       "2010-05-25  10.47  11.05  10.42  11.02  137858600   9.024753\n",
       "2010-06-24  10.99  11.03  10.64  10.78   74449200   8.828206\n",
       "2010-06-25  10.75  10.77  10.42  10.75  148168400   8.803638\n",
       "2010-06-28  10.72  10.76  10.43  10.43   57994700   8.541577\n",
       "2010-06-29  10.12  10.14   9.75   9.88  124866100   8.091158\n",
       "2010-06-30  10.10  10.52  10.04  10.08  182831300   8.254946\n",
       "2010-07-01  10.23  10.63  10.02  10.57  153383000   8.656228\n",
       "2010-07-02  10.58  10.63  10.15  10.28   71607100   8.418735\n",
       "2010-07-06  10.42  10.57  10.04  10.16   81246900   8.320462\n",
       "2010-07-07  10.21  10.63  10.17  10.59   72565300   8.672607\n",
       "2010-07-08  10.72  10.83  10.45  10.62   62355500   8.697175\n",
       "2010-07-09  10.58  10.94  10.54  10.85   50732700   8.885533\n",
       "2010-07-12  10.84  11.13  10.83  11.10   56354100   9.090269\n",
       "2010-01-04  10.17  10.28  10.05  10.28   60855800   8.418735\n",
       "2010-01-05  10.45  11.24  10.40  10.96  215620200   8.975616\n",
       "2010-01-25  10.73  11.10  10.61  11.03  121621500   9.032942\n",
       "2010-02-05  10.97  11.11  10.49  10.91  181535200   8.934669\n",
       "2010-02-12  10.92  11.18  10.85  11.12   69465400   9.106647\n",
       "2010-05-21  10.25  11.30  10.17  11.26  174455100   9.221299\n",
       "2010-05-25  10.47  11.05  10.42  11.02  137858600   9.024753\n",
       "2010-06-24  10.99  11.03  10.64  10.78   74449200   8.828206\n",
       "2010-06-25  10.75  10.77  10.42  10.75  148168400   8.803638\n",
       "2010-06-28  10.72  10.76  10.43  10.43   57994700   8.541577\n",
       "2010-06-29  10.12  10.14   9.75   9.88  124866100   8.091158\n",
       "2010-06-30  10.10  10.52  10.04  10.08  182831300   8.254946\n",
       "2010-07-01  10.23  10.63  10.02  10.57  153383000   8.656228\n",
       "2010-07-02  10.58  10.63  10.15  10.28   71607100   8.418735\n",
       "2010-07-06  10.42  10.57  10.04  10.16   81246900   8.320462\n",
       "2010-07-07  10.21  10.63  10.17  10.59   72565300   8.672607\n",
       "2010-07-08  10.72  10.83  10.45  10.62   62355500   8.697175\n",
       "2010-07-09  10.58  10.94  10.54  10.85   50732700   8.885533\n",
       "2010-07-12  10.84  11.13  10.83  11.10   56354100   9.090269"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.select(\"yahoo_indexed\", \"index<=Timestamp('2011-01-01') & Open>10 & Open<11\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now also \"Open\" and \"Close\" are indexed!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Storing own meta data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "storer = store.get_storer('yahoo')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attributes are stored by *pickle* so many Python objects are possible:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "storer.attrs.my_attribute = dict(A = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store = pd.HDFStore(\"example-pandas.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'A': 10}"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.get_storer(\"yahoo\").attrs.my_attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show stored object in hdfview!\n",
    "Own format on top of HDF5 using pytables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "Only a subset HDF5 features usable but quite easy to use with pandas including indexed search on disk.\n",
    "\n",
    "For more than two-dimensions, *xarray* is worth a look.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.Dataset>\n",
       "Dimensions:    (Date: 1542)\n",
       "Coordinates:\n",
       "  * Date       (Date) datetime64[ns] 2010-01-04 2010-01-05 2010-01-06 ...\n",
       "Data variables:\n",
       "    Open       (Date) float64 10.17 10.45 11.21 11.46 11.67 11.9 11.98 11.91 ...\n",
       "    High       (Date) float64 10.28 11.24 11.46 11.69 11.74 12.14 12.03 ...\n",
       "    Low        (Date) float64 10.05 10.4 11.13 11.32 11.46 11.78 11.72 11.47 ...\n",
       "    Close      (Date) float64 10.28 10.96 11.37 11.66 11.69 12.11 11.87 ...\n",
       "    Volume     (Date) int64 60855800 215620200 200070600 130201700 130463000 ...\n",
       "    Adj Close  (Date) float64 8.419 8.976 9.311 9.549 9.573 9.917 9.721 ..."
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.to_xarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}