Assignment+3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.5** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-data-analysis/resources/0dhYG) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 3 - More Pandas\n",
    "This assignment requires more individual learning then the last one did - you are encouraged to check out the [pandas documentation](http://pandas.pydata.org/pandas-docs/stable/) to find functions or methods you might not have used yet, or ask questions on [Stack Overflow](http://stackoverflow.com/) and tag them as pandas and python related. And of course, the discussion forums are open for interaction with your peers and the course staff."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1 (20%)\n",
    "Load the energy data from the file `Energy Indicators.xls`, which is a list of indicators of [energy supply and renewable electricity production](Energy%20Indicators.xls) from the [United Nations](http://unstats.un.org/unsd/environment/excel_file_tables/2013/Energy%20Indicators.xls) for the year 2013, and should be put into a DataFrame with the variable name of **energy**.\n",
    "\n",
    "Keep in mind that this is an Excel file, and not a comma separated values file. Also, make sure to exclude the footer and header information from the datafile. The first two columns are unneccessary, so you should get rid of them, and you should change the column labels so that the columns are:\n",
    "\n",
    "`['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']`\n",
    "\n",
    "Convert `Energy Supply` to gigajoules (there are 1,000,000 gigajoules in a petajoule). For all countries which have missing data (e.g. data with \"...\") make sure this is reflected as `np.NaN` values.\n",
    "\n",
    "Rename the following list of countries (for use in later questions):\n",
    "\n",
    "```\"Republic of Korea\": \"South Korea\",\n",
    "\"United States of America\": \"United States\",\n",
    "\"United Kingdom of Great Britain and Northern Ireland\": \"United Kingdom\",\n",
    "\"China, Hong Kong Special Administrative Region\": \"Hong Kong\"```\n",
    "\n",
    "There are also several countries with numbers and/or parenthesis in their name. Be sure to remove these, \n",
    "\n",
    "e.g. \n",
    "\n",
    "`'Bolivia (Plurinational State of)'` should be `'Bolivia'`, \n",
    "\n",
    "`'Switzerland17'` should be `'Switzerland'`.\n",
    "\n",
    "<br>\n",
    "\n",
    "Next, load the GDP data from the file `world_bank.csv`, which is a csv containing countries' GDP from 1960 to 2015 from [World Bank](http://data.worldbank.org/indicator/NY.GDP.MKTP.CD). Call this DataFrame **GDP**. \n",
    "\n",
    "Make sure to skip the header, and rename the following list of countries:\n",
    "\n",
    "```\"Korea, Rep.\": \"South Korea\", \n",
    "\"Iran, Islamic Rep.\": \"Iran\",\n",
    "\"Hong Kong SAR, China\": \"Hong Kong\"```\n",
    "\n",
    "<br>\n",
    "\n",
    "Finally, load the [Sciamgo Journal and Country Rank data for Energy Engineering and Power Technology](http://www.scimagojr.com/countryrank.php?category=2102) from the file `scimagojr-3.xlsx`, which ranks countries based on their journal contributions in the aforementioned area. Call this DataFrame **ScimEn**.\n",
    "\n",
    "Join the three datasets: GDP, Energy, and ScimEn into a new dataset (using the intersection of country names). Use only the last 10 years (2006-2015) of GDP data and only the top 15 countries by Scimagojr 'Rank' (Rank 1 through 15). \n",
    "\n",
    "The index of this DataFrame should be the name of the country, and the columns should be ['Rank', 'Documents', 'Citable documents', 'Citations', 'Self-citations',\n",
    "       'Citations per document', 'H index', 'Energy Supply',\n",
    "       'Energy Supply per Capita', '% Renewable', '2006', '2007', '2008',\n",
    "       '2009', '2010', '2011', '2012', '2013', '2014', '2015'].\n",
    "\n",
    "*This function should return a DataFrame with 20 columns and 15 entries.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import re\n",
    "from numbers import Number as nb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 1</th>\n",
       "      <th>Unnamed: 2</th>\n",
       "      <th>Petajoules</th>\n",
       "      <th>Gigajoules</th>\n",
       "      <th>%</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>321</td>\n",
       "      <td>10</td>\n",
       "      <td>78.669280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Albania</td>\n",
       "      <td>Albania</td>\n",
       "      <td>102</td>\n",
       "      <td>35</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Algeria</td>\n",
       "      <td>Algeria</td>\n",
       "      <td>1959</td>\n",
       "      <td>51</td>\n",
       "      <td>0.551010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>American Samoa</td>\n",
       "      <td>American Samoa</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>0.641026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Andorra</td>\n",
       "      <td>Andorra</td>\n",
       "      <td>9</td>\n",
       "      <td>121</td>\n",
       "      <td>88.695650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Angola</td>\n",
       "      <td>Angola</td>\n",
       "      <td>642</td>\n",
       "      <td>27</td>\n",
       "      <td>70.909090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Anguilla</td>\n",
       "      <td>Anguilla</td>\n",
       "      <td>2</td>\n",
       "      <td>136</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Antigua and Barbuda</td>\n",
       "      <td>Antigua and Barbuda</td>\n",
       "      <td>8</td>\n",
       "      <td>84</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Argentina</td>\n",
       "      <td>Argentina</td>\n",
       "      <td>3378</td>\n",
       "      <td>79</td>\n",
       "      <td>24.064520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Armenia</td>\n",
       "      <td>Armenia</td>\n",
       "      <td>143</td>\n",
       "      <td>48</td>\n",
       "      <td>28.236060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Aruba</td>\n",
       "      <td>Aruba</td>\n",
       "      <td>12</td>\n",
       "      <td>120</td>\n",
       "      <td>14.870690</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Australia</td>\n",
       "      <td>Australia1</td>\n",
       "      <td>5386</td>\n",
       "      <td>231</td>\n",
       "      <td>11.810810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Austria</td>\n",
       "      <td>Austria</td>\n",
       "      <td>1391</td>\n",
       "      <td>164</td>\n",
       "      <td>72.452820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Azerbaijan</td>\n",
       "      <td>Azerbaijan</td>\n",
       "      <td>567</td>\n",
       "      <td>60</td>\n",
       "      <td>6.384345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Bahamas</td>\n",
       "      <td>Bahamas</td>\n",
       "      <td>45</td>\n",
       "      <td>118</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Bahrain</td>\n",
       "      <td>Bahrain</td>\n",
       "      <td>574</td>\n",
       "      <td>425</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Bangladesh</td>\n",
       "      <td>Bangladesh</td>\n",
       "      <td>1625</td>\n",
       "      <td>10</td>\n",
       "      <td>1.966329</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Barbados</td>\n",
       "      <td>Barbados</td>\n",
       "      <td>19</td>\n",
       "      <td>69</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Belarus</td>\n",
       "      <td>Belarus</td>\n",
       "      <td>1142</td>\n",
       "      <td>120</td>\n",
       "      <td>0.463389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Belgium</td>\n",
       "      <td>Belgium</td>\n",
       "      <td>2338</td>\n",
       "      <td>210</td>\n",
       "      <td>9.959739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Belize</td>\n",
       "      <td>Belize</td>\n",
       "      <td>13</td>\n",
       "      <td>39</td>\n",
       "      <td>64.690030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Benin</td>\n",
       "      <td>Benin</td>\n",
       "      <td>170</td>\n",
       "      <td>16</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Bermuda</td>\n",
       "      <td>Bermuda</td>\n",
       "      <td>6</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Bhutan</td>\n",
       "      <td>Bhutan</td>\n",
       "      <td>63</td>\n",
       "      <td>83</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Bolivia (Plurinational State of)</td>\n",
       "      <td>Bolivia (Plurinational State of)</td>\n",
       "      <td>336</td>\n",
       "      <td>32</td>\n",
       "      <td>31.477120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Bonaire, Sint Eustatius and Saba</td>\n",
       "      <td>Bonaire, Sint Eustatius and Saba</td>\n",
       "      <td>5</td>\n",
       "      <td>213</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Bosnia and Herzegovina</td>\n",
       "      <td>Bosnia and Herzegovina</td>\n",
       "      <td>266</td>\n",
       "      <td>70</td>\n",
       "      <td>41.464670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Botswana</td>\n",
       "      <td>Botswana</td>\n",
       "      <td>86</td>\n",
       "      <td>39</td>\n",
       "      <td>0.131406</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Brazil</td>\n",
       "      <td>Brazil</td>\n",
       "      <td>12149</td>\n",
       "      <td>59</td>\n",
       "      <td>69.648030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>NaN</td>\n",
       "      <td>British Virgin Islands</td>\n",
       "      <td>British Virgin Islands</td>\n",
       "      <td>2</td>\n",
       "      <td>85</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Brunei Darussalam</td>\n",
       "      <td>Brunei Darussalam</td>\n",
       "      <td>127</td>\n",
       "      <td>310</td>\n",
       "      <td>0.045434</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Bulgaria</td>\n",
       "      <td>Bulgaria</td>\n",
       "      <td>703</td>\n",
       "      <td>97</td>\n",
       "      <td>17.234610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Burkina Faso</td>\n",
       "      <td>Burkina Faso</td>\n",
       "      <td>164</td>\n",
       "      <td>10</td>\n",
       "      <td>14.500680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Burundi</td>\n",
       "      <td>Burundi</td>\n",
       "      <td>58</td>\n",
       "      <td>6</td>\n",
       "      <td>86.792450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Cabo Verde</td>\n",
       "      <td>Cabo Verde</td>\n",
       "      <td>7</td>\n",
       "      <td>15</td>\n",
       "      <td>21.227620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Cambodia</td>\n",
       "      <td>Cambodia</td>\n",
       "      <td>250</td>\n",
       "      <td>17</td>\n",
       "      <td>57.311590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Cameroon</td>\n",
       "      <td>Cameroon</td>\n",
       "      <td>307</td>\n",
       "      <td>14</td>\n",
       "      <td>70.886260</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Canada</td>\n",
       "      <td>Canada</td>\n",
       "      <td>10431</td>\n",
       "      <td>296</td>\n",
       "      <td>61.945430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Cayman Islands</td>\n",
       "      <td>Cayman Islands</td>\n",
       "      <td>8</td>\n",
       "      <td>131</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Central African Republic</td>\n",
       "      <td>Central African Republic</td>\n",
       "      <td>23</td>\n",
       "      <td>5</td>\n",
       "      <td>85.635360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Chad</td>\n",
       "      <td>Chad</td>\n",
       "      <td>77</td>\n",
       "      <td>6</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Chile</td>\n",
       "      <td>Chile</td>\n",
       "      <td>1613</td>\n",
       "      <td>92</td>\n",
       "      <td>28.006570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>NaN</td>\n",
       "      <td>China</td>\n",
       "      <td>China2</td>\n",
       "      <td>127191</td>\n",
       "      <td>93</td>\n",
       "      <td>19.754910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>NaN</td>\n",
       "      <td>China, Hong Kong Special Administrative Region</td>\n",
       "      <td>China, Hong Kong Special Administrative Region3</td>\n",
       "      <td>585</td>\n",
       "      <td>82</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>NaN</td>\n",
       "      <td>China, Macao Special Administrative Region</td>\n",
       "      <td>China, Macao Special Administrative Region4</td>\n",
       "      <td>47</td>\n",
       "      <td>83</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Colombia</td>\n",
       "      <td>Colombia</td>\n",
       "      <td>1513</td>\n",
       "      <td>32</td>\n",
       "      <td>63.002840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Comoros</td>\n",
       "      <td>Comoros</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>11.627910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Congo</td>\n",
       "      <td>Congo</td>\n",
       "      <td>102</td>\n",
       "      <td>23</td>\n",
       "      <td>57.748870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Cook Islands</td>\n",
       "      <td>Cook Islands</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Costa Rica</td>\n",
       "      <td>Costa Rica</td>\n",
       "      <td>222</td>\n",
       "      <td>47</td>\n",
       "      <td>88.736420</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Unnamed: 0                                      Unnamed: 1  \\\n",
       "0          NaN                                     Afghanistan   \n",
       "1          NaN                                         Albania   \n",
       "2          NaN                                         Algeria   \n",
       "3          NaN                                  American Samoa   \n",
       "4          NaN                                         Andorra   \n",
       "5          NaN                                          Angola   \n",
       "6          NaN                                        Anguilla   \n",
       "7          NaN                             Antigua and Barbuda   \n",
       "8          NaN                                       Argentina   \n",
       "9          NaN                                         Armenia   \n",
       "10         NaN                                           Aruba   \n",
       "11         NaN                                       Australia   \n",
       "12         NaN                                         Austria   \n",
       "13         NaN                                      Azerbaijan   \n",
       "14         NaN                                         Bahamas   \n",
       "15         NaN                                         Bahrain   \n",
       "16         NaN                                      Bangladesh   \n",
       "17         NaN                                        Barbados   \n",
       "18         NaN                                         Belarus   \n",
       "19         NaN                                         Belgium   \n",
       "20         NaN                                          Belize   \n",
       "21         NaN                                           Benin   \n",
       "22         NaN                                         Bermuda   \n",
       "23         NaN                                          Bhutan   \n",
       "24         NaN                Bolivia (Plurinational State of)   \n",
       "25         NaN                Bonaire, Sint Eustatius and Saba   \n",
       "26         NaN                          Bosnia and Herzegovina   \n",
       "27         NaN                                        Botswana   \n",
       "28         NaN                                          Brazil   \n",
       "29         NaN                          British Virgin Islands   \n",
       "30         NaN                               Brunei Darussalam   \n",
       "31         NaN                                        Bulgaria   \n",
       "32         NaN                                    Burkina Faso   \n",
       "33         NaN                                         Burundi   \n",
       "34         NaN                                      Cabo Verde   \n",
       "35         NaN                                        Cambodia   \n",
       "36         NaN                                        Cameroon   \n",
       "37         NaN                                          Canada   \n",
       "38         NaN                                  Cayman Islands   \n",
       "39         NaN                        Central African Republic   \n",
       "40         NaN                                            Chad   \n",
       "41         NaN                                           Chile   \n",
       "42         NaN                                           China   \n",
       "43         NaN  China, Hong Kong Special Administrative Region   \n",
       "44         NaN      China, Macao Special Administrative Region   \n",
       "45         NaN                                        Colombia   \n",
       "46         NaN                                         Comoros   \n",
       "47         NaN                                           Congo   \n",
       "48         NaN                                    Cook Islands   \n",
       "49         NaN                                      Costa Rica   \n",
       "\n",
       "                                         Unnamed: 2 Petajoules Gigajoules  \\\n",
       "0                                       Afghanistan        321         10   \n",
       "1                                           Albania        102         35   \n",
       "2                                           Algeria       1959         51   \n",
       "3                                    American Samoa        ...        ...   \n",
       "4                                           Andorra          9        121   \n",
       "5                                            Angola        642         27   \n",
       "6                                          Anguilla          2        136   \n",
       "7                               Antigua and Barbuda          8         84   \n",
       "8                                         Argentina       3378         79   \n",
       "9                                           Armenia        143         48   \n",
       "10                                            Aruba         12        120   \n",
       "11                                       Australia1       5386        231   \n",
       "12                                          Austria       1391        164   \n",
       "13                                       Azerbaijan        567         60   \n",
       "14                                          Bahamas         45        118   \n",
       "15                                          Bahrain        574        425   \n",
       "16                                       Bangladesh       1625         10   \n",
       "17                                         Barbados         19         69   \n",
       "18                                          Belarus       1142        120   \n",
       "19                                          Belgium       2338        210   \n",
       "20                                           Belize         13         39   \n",
       "21                                            Benin        170         16   \n",
       "22                                          Bermuda          6         88   \n",
       "23                                           Bhutan         63         83   \n",
       "24                 Bolivia (Plurinational State of)        336         32   \n",
       "25                 Bonaire, Sint Eustatius and Saba          5        213   \n",
       "26                           Bosnia and Herzegovina        266         70   \n",
       "27                                         Botswana         86         39   \n",
       "28                                           Brazil      12149         59   \n",
       "29                           British Virgin Islands          2         85   \n",
       "30                                Brunei Darussalam        127        310   \n",
       "31                                         Bulgaria        703         97   \n",
       "32                                     Burkina Faso        164         10   \n",
       "33                                          Burundi         58          6   \n",
       "34                                       Cabo Verde          7         15   \n",
       "35                                         Cambodia        250         17   \n",
       "36                                         Cameroon        307         14   \n",
       "37                                           Canada      10431        296   \n",
       "38                                   Cayman Islands          8        131   \n",
       "39                         Central African Republic         23          5   \n",
       "40                                             Chad         77          6   \n",
       "41                                            Chile       1613         92   \n",
       "42                                           China2     127191         93   \n",
       "43  China, Hong Kong Special Administrative Region3        585         82   \n",
       "44      China, Macao Special Administrative Region4         47         83   \n",
       "45                                         Colombia       1513         32   \n",
       "46                                          Comoros          5          7   \n",
       "47                                            Congo        102         23   \n",
       "48                                     Cook Islands          1         48   \n",
       "49                                       Costa Rica        222         47   \n",
       "\n",
       "             %  \n",
       "0    78.669280  \n",
       "1   100.000000  \n",
       "2     0.551010  \n",
       "3     0.641026  \n",
       "4    88.695650  \n",
       "5    70.909090  \n",
       "6     0.000000  \n",
       "7     0.000000  \n",
       "8    24.064520  \n",
       "9    28.236060  \n",
       "10   14.870690  \n",
       "11   11.810810  \n",
       "12   72.452820  \n",
       "13    6.384345  \n",
       "14    0.000000  \n",
       "15    0.000000  \n",
       "16    1.966329  \n",
       "17    0.000000  \n",
       "18    0.463389  \n",
       "19    9.959739  \n",
       "20   64.690030  \n",
       "21    0.000000  \n",
       "22    0.000000  \n",
       "23  100.000000  \n",
       "24   31.477120  \n",
       "25    0.000000  \n",
       "26   41.464670  \n",
       "27    0.131406  \n",
       "28   69.648030  \n",
       "29    0.000000  \n",
       "30    0.045434  \n",
       "31   17.234610  \n",
       "32   14.500680  \n",
       "33   86.792450  \n",
       "34   21.227620  \n",
       "35   57.311590  \n",
       "36   70.886260  \n",
       "37   61.945430  \n",
       "38    0.000000  \n",
       "39   85.635360  \n",
       "40    0.000000  \n",
       "41   28.006570  \n",
       "42   19.754910  \n",
       "43    0.000000  \n",
       "44    0.000000  \n",
       "45   63.002840  \n",
       "46   11.627910  \n",
       "47   57.748870  \n",
       "48    0.000000  \n",
       "49   88.736420  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "energy = pd.read_excel('Energy Indicators.xls', skiprows=[i for i in range(17)], skipfooter=32)\n",
    "GDP = pd.read_csv(\"world_bank.csv\", skiprows=[0, 1, 2, 3])\n",
    "energy.head(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "umich_part_id": "009",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Energy Supply</th>\n",
       "      <th>Energy Supply per Capita</th>\n",
       "      <th>% Renewable</th>\n",
       "      <th>2006</th>\n",
       "      <th>2007</th>\n",
       "      <th>2008</th>\n",
       "      <th>2009</th>\n",
       "      <th>2010</th>\n",
       "      <th>2011</th>\n",
       "      <th>2012</th>\n",
       "      <th>2013</th>\n",
       "      <th>2014</th>\n",
       "      <th>2015</th>\n",
       "      <th>Rank</th>\n",
       "      <th>Documents</th>\n",
       "      <th>Citable documents</th>\n",
       "      <th>Citations</th>\n",
       "      <th>Self-citations</th>\n",
       "      <th>Citations per document</th>\n",
       "      <th>H index</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>China</th>\n",
       "      <td>1.271910e+11</td>\n",
       "      <td>93</td>\n",
       "      <td>19.754910</td>\n",
       "      <td>3.992331e+12</td>\n",
       "      <td>4.559041e+12</td>\n",
       "      <td>4.997775e+12</td>\n",
       "      <td>5.459247e+12</td>\n",
       "      <td>6.039659e+12</td>\n",
       "      <td>6.612490e+12</td>\n",
       "      <td>7.124978e+12</td>\n",
       "      <td>7.672448e+12</td>\n",
       "      <td>8.230121e+12</td>\n",
       "      <td>8.797999e+12</td>\n",
       "      <td>1</td>\n",
       "      <td>127050</td>\n",
       "      <td>126767</td>\n",
       "      <td>597237</td>\n",
       "      <td>411683</td>\n",
       "      <td>4.70</td>\n",
       "      <td>138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>United States</th>\n",
       "      <td>9.083800e+10</td>\n",
       "      <td>286</td>\n",
       "      <td>11.570980</td>\n",
       "      <td>1.479230e+13</td>\n",
       "      <td>1.505540e+13</td>\n",
       "      <td>1.501149e+13</td>\n",
       "      <td>1.459484e+13</td>\n",
       "      <td>1.496437e+13</td>\n",
       "      <td>1.520402e+13</td>\n",
       "      <td>1.554216e+13</td>\n",
       "      <td>1.577367e+13</td>\n",
       "      <td>1.615662e+13</td>\n",
       "      <td>1.654857e+13</td>\n",
       "      <td>2</td>\n",
       "      <td>96661</td>\n",
       "      <td>94747</td>\n",
       "      <td>792274</td>\n",
       "      <td>265436</td>\n",
       "      <td>8.20</td>\n",
       "      <td>230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Japan</th>\n",
       "      <td>1.898400e+10</td>\n",
       "      <td>149</td>\n",
       "      <td>10.232820</td>\n",
       "      <td>5.496542e+12</td>\n",
       "      <td>5.617036e+12</td>\n",
       "      <td>5.558527e+12</td>\n",
       "      <td>5.251308e+12</td>\n",
       "      <td>5.498718e+12</td>\n",
       "      <td>5.473738e+12</td>\n",
       "      <td>5.569102e+12</td>\n",
       "      <td>5.644659e+12</td>\n",
       "      <td>5.642884e+12</td>\n",
       "      <td>5.669563e+12</td>\n",
       "      <td>3</td>\n",
       "      <td>30504</td>\n",
       "      <td>30287</td>\n",
       "      <td>223024</td>\n",
       "      <td>61554</td>\n",
       "      <td>7.31</td>\n",
       "      <td>134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>United Kingdom</th>\n",
       "      <td>7.920000e+09</td>\n",
       "      <td>124</td>\n",
       "      <td>10.600470</td>\n",
       "      <td>2.419631e+12</td>\n",
       "      <td>2.482203e+12</td>\n",
       "      <td>2.470614e+12</td>\n",
       "      <td>2.367048e+12</td>\n",
       "      <td>2.403504e+12</td>\n",
       "      <td>2.450911e+12</td>\n",
       "      <td>2.479809e+12</td>\n",
       "      <td>2.533370e+12</td>\n",
       "      <td>2.605643e+12</td>\n",
       "      <td>2.666333e+12</td>\n",
       "      <td>4</td>\n",
       "      <td>20944</td>\n",
       "      <td>20357</td>\n",
       "      <td>206091</td>\n",
       "      <td>37874</td>\n",
       "      <td>9.84</td>\n",
       "      <td>139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Russian Federation</th>\n",
       "      <td>3.070900e+10</td>\n",
       "      <td>214</td>\n",
       "      <td>17.288680</td>\n",
       "      <td>1.385793e+12</td>\n",
       "      <td>1.504071e+12</td>\n",
       "      <td>1.583004e+12</td>\n",
       "      <td>1.459199e+12</td>\n",
       "      <td>1.524917e+12</td>\n",
       "      <td>1.589943e+12</td>\n",
       "      <td>1.645876e+12</td>\n",
       "      <td>1.666934e+12</td>\n",
       "      <td>1.678709e+12</td>\n",
       "      <td>1.616149e+12</td>\n",
       "      <td>5</td>\n",
       "      <td>18534</td>\n",
       "      <td>18301</td>\n",
       "      <td>34266</td>\n",
       "      <td>12422</td>\n",
       "      <td>1.85</td>\n",
       "      <td>57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Canada</th>\n",
       "      <td>1.043100e+10</td>\n",
       "      <td>296</td>\n",
       "      <td>61.945430</td>\n",
       "      <td>1.564469e+12</td>\n",
       "      <td>1.596740e+12</td>\n",
       "      <td>1.612713e+12</td>\n",
       "      <td>1.565145e+12</td>\n",
       "      <td>1.613406e+12</td>\n",
       "      <td>1.664087e+12</td>\n",
       "      <td>1.693133e+12</td>\n",
       "      <td>1.730688e+12</td>\n",
       "      <td>1.773486e+12</td>\n",
       "      <td>1.792609e+12</td>\n",
       "      <td>6</td>\n",
       "      <td>17899</td>\n",
       "      <td>17620</td>\n",
       "      <td>215003</td>\n",
       "      <td>40930</td>\n",
       "      <td>12.01</td>\n",
       "      <td>149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Germany</th>\n",
       "      <td>1.326100e+10</td>\n",
       "      <td>165</td>\n",
       "      <td>17.901530</td>\n",
       "      <td>3.332891e+12</td>\n",
       "      <td>3.441561e+12</td>\n",
       "      <td>3.478809e+12</td>\n",
       "      <td>3.283340e+12</td>\n",
       "      <td>3.417298e+12</td>\n",
       "      <td>3.542371e+12</td>\n",
       "      <td>3.556724e+12</td>\n",
       "      <td>3.567317e+12</td>\n",
       "      <td>3.624386e+12</td>\n",
       "      <td>3.685556e+12</td>\n",
       "      <td>7</td>\n",
       "      <td>17027</td>\n",
       "      <td>16831</td>\n",
       "      <td>140566</td>\n",
       "      <td>27426</td>\n",
       "      <td>8.26</td>\n",
       "      <td>126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>India</th>\n",
       "      <td>3.319500e+10</td>\n",
       "      <td>26</td>\n",
       "      <td>14.969080</td>\n",
       "      <td>1.265894e+12</td>\n",
       "      <td>1.374865e+12</td>\n",
       "      <td>1.428361e+12</td>\n",
       "      <td>1.549483e+12</td>\n",
       "      <td>1.708459e+12</td>\n",
       "      <td>1.821872e+12</td>\n",
       "      <td>1.924235e+12</td>\n",
       "      <td>2.051982e+12</td>\n",
       "      <td>2.200617e+12</td>\n",
       "      <td>2.367206e+12</td>\n",
       "      <td>8</td>\n",
       "      <td>15005</td>\n",
       "      <td>14841</td>\n",
       "      <td>128763</td>\n",
       "      <td>37209</td>\n",
       "      <td>8.58</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>France</th>\n",
       "      <td>1.059700e+10</td>\n",
       "      <td>166</td>\n",
       "      <td>17.020280</td>\n",
       "      <td>2.607840e+12</td>\n",
       "      <td>2.669424e+12</td>\n",
       "      <td>2.674637e+12</td>\n",
       "      <td>2.595967e+12</td>\n",
       "      <td>2.646995e+12</td>\n",
       "      <td>2.702032e+12</td>\n",
       "      <td>2.706968e+12</td>\n",
       "      <td>2.722567e+12</td>\n",
       "      <td>2.729632e+12</td>\n",
       "      <td>2.761185e+12</td>\n",
       "      <td>9</td>\n",
       "      <td>13153</td>\n",
       "      <td>12973</td>\n",
       "      <td>130632</td>\n",
       "      <td>28601</td>\n",
       "      <td>9.93</td>\n",
       "      <td>114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>South Korea</th>\n",
       "      <td>1.100700e+10</td>\n",
       "      <td>221</td>\n",
       "      <td>2.279353</td>\n",
       "      <td>9.410199e+11</td>\n",
       "      <td>9.924316e+11</td>\n",
       "      <td>1.020510e+12</td>\n",
       "      <td>1.027730e+12</td>\n",
       "      <td>1.094499e+12</td>\n",
       "      <td>1.134796e+12</td>\n",
       "      <td>1.160809e+12</td>\n",
       "      <td>1.194429e+12</td>\n",
       "      <td>1.234340e+12</td>\n",
       "      <td>1.266580e+12</td>\n",
       "      <td>10</td>\n",
       "      <td>11983</td>\n",
       "      <td>11923</td>\n",
       "      <td>114675</td>\n",
       "      <td>22595</td>\n",
       "      <td>9.57</td>\n",
       "      <td>104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Italy</th>\n",
       "      <td>6.530000e+09</td>\n",
       "      <td>109</td>\n",
       "      <td>33.667230</td>\n",
       "      <td>2.202170e+12</td>\n",
       "      <td>2.234627e+12</td>\n",
       "      <td>2.211154e+12</td>\n",
       "      <td>2.089938e+12</td>\n",
       "      <td>2.125185e+12</td>\n",
       "      <td>2.137439e+12</td>\n",
       "      <td>2.077184e+12</td>\n",
       "      <td>2.040871e+12</td>\n",
       "      <td>2.033868e+12</td>\n",
       "      <td>2.049316e+12</td>\n",
       "      <td>11</td>\n",
       "      <td>10964</td>\n",
       "      <td>10794</td>\n",
       "      <td>111850</td>\n",
       "      <td>26661</td>\n",
       "      <td>10.20</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spain</th>\n",
       "      <td>4.923000e+09</td>\n",
       "      <td>106</td>\n",
       "      <td>37.968590</td>\n",
       "      <td>1.414823e+12</td>\n",
       "      <td>1.468146e+12</td>\n",
       "      <td>1.484530e+12</td>\n",
       "      <td>1.431475e+12</td>\n",
       "      <td>1.431673e+12</td>\n",
       "      <td>1.417355e+12</td>\n",
       "      <td>1.380216e+12</td>\n",
       "      <td>1.357139e+12</td>\n",
       "      <td>1.375605e+12</td>\n",
       "      <td>1.419821e+12</td>\n",
       "      <td>12</td>\n",
       "      <td>9428</td>\n",
       "      <td>9330</td>\n",
       "      <td>123336</td>\n",
       "      <td>23964</td>\n",
       "      <td>13.08</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iran</th>\n",
       "      <td>9.172000e+09</td>\n",
       "      <td>119</td>\n",
       "      <td>5.707721</td>\n",
       "      <td>3.895523e+11</td>\n",
       "      <td>4.250646e+11</td>\n",
       "      <td>4.289909e+11</td>\n",
       "      <td>4.389208e+11</td>\n",
       "      <td>4.677902e+11</td>\n",
       "      <td>4.853309e+11</td>\n",
       "      <td>4.532569e+11</td>\n",
       "      <td>4.445926e+11</td>\n",
       "      <td>4.639027e+11</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13</td>\n",
       "      <td>8896</td>\n",
       "      <td>8819</td>\n",
       "      <td>57470</td>\n",
       "      <td>19125</td>\n",
       "      <td>6.46</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Australia</th>\n",
       "      <td>5.386000e+09</td>\n",
       "      <td>231</td>\n",
       "      <td>11.810810</td>\n",
       "      <td>1.021939e+12</td>\n",
       "      <td>1.060340e+12</td>\n",
       "      <td>1.099644e+12</td>\n",
       "      <td>1.119654e+12</td>\n",
       "      <td>1.142251e+12</td>\n",
       "      <td>1.169431e+12</td>\n",
       "      <td>1.211913e+12</td>\n",
       "      <td>1.241484e+12</td>\n",
       "      <td>1.272520e+12</td>\n",
       "      <td>1.301251e+12</td>\n",
       "      <td>14</td>\n",
       "      <td>8831</td>\n",
       "      <td>8725</td>\n",
       "      <td>90765</td>\n",
       "      <td>15606</td>\n",
       "      <td>10.28</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Brazil</th>\n",
       "      <td>1.214900e+10</td>\n",
       "      <td>59</td>\n",
       "      <td>69.648030</td>\n",
       "      <td>1.845080e+12</td>\n",
       "      <td>1.957118e+12</td>\n",
       "      <td>2.056809e+12</td>\n",
       "      <td>2.054215e+12</td>\n",
       "      <td>2.208872e+12</td>\n",
       "      <td>2.295245e+12</td>\n",
       "      <td>2.339209e+12</td>\n",
       "      <td>2.409740e+12</td>\n",
       "      <td>2.412231e+12</td>\n",
       "      <td>2.319423e+12</td>\n",
       "      <td>15</td>\n",
       "      <td>8668</td>\n",
       "      <td>8596</td>\n",
       "      <td>60702</td>\n",
       "      <td>14396</td>\n",
       "      <td>7.00</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Energy Supply Energy Supply per Capita  % Renewable  \\\n",
       "Country                                                                   \n",
       "China                1.271910e+11                       93    19.754910   \n",
       "United States        9.083800e+10                      286    11.570980   \n",
       "Japan                1.898400e+10                      149    10.232820   \n",
       "United Kingdom       7.920000e+09                      124    10.600470   \n",
       "Russian Federation   3.070900e+10                      214    17.288680   \n",
       "Canada               1.043100e+10                      296    61.945430   \n",
       "Germany              1.326100e+10                      165    17.901530   \n",
       "India                3.319500e+10                       26    14.969080   \n",
       "France               1.059700e+10                      166    17.020280   \n",
       "South Korea          1.100700e+10                      221     2.279353   \n",
       "Italy                6.530000e+09                      109    33.667230   \n",
       "Spain                4.923000e+09                      106    37.968590   \n",
       "Iran                 9.172000e+09                      119     5.707721   \n",
       "Australia            5.386000e+09                      231    11.810810   \n",
       "Brazil               1.214900e+10                       59    69.648030   \n",
       "\n",
       "                            2006          2007          2008          2009  \\\n",
       "Country                                                                      \n",
       "China               3.992331e+12  4.559041e+12  4.997775e+12  5.459247e+12   \n",
       "United States       1.479230e+13  1.505540e+13  1.501149e+13  1.459484e+13   \n",
       "Japan               5.496542e+12  5.617036e+12  5.558527e+12  5.251308e+12   \n",
       "United Kingdom      2.419631e+12  2.482203e+12  2.470614e+12  2.367048e+12   \n",
       "Russian Federation  1.385793e+12  1.504071e+12  1.583004e+12  1.459199e+12   \n",
       "Canada              1.564469e+12  1.596740e+12  1.612713e+12  1.565145e+12   \n",
       "Germany             3.332891e+12  3.441561e+12  3.478809e+12  3.283340e+12   \n",
       "India               1.265894e+12  1.374865e+12  1.428361e+12  1.549483e+12   \n",
       "France              2.607840e+12  2.669424e+12  2.674637e+12  2.595967e+12   \n",
       "South Korea         9.410199e+11  9.924316e+11  1.020510e+12  1.027730e+12   \n",
       "Italy               2.202170e+12  2.234627e+12  2.211154e+12  2.089938e+12   \n",
       "Spain               1.414823e+12  1.468146e+12  1.484530e+12  1.431475e+12   \n",
       "Iran                3.895523e+11  4.250646e+11  4.289909e+11  4.389208e+11   \n",
       "Australia           1.021939e+12  1.060340e+12  1.099644e+12  1.119654e+12   \n",
       "Brazil              1.845080e+12  1.957118e+12  2.056809e+12  2.054215e+12   \n",
       "\n",
       "                            2010          2011          2012          2013  \\\n",
       "Country                                                                      \n",
       "China               6.039659e+12  6.612490e+12  7.124978e+12  7.672448e+12   \n",
       "United States       1.496437e+13  1.520402e+13  1.554216e+13  1.577367e+13   \n",
       "Japan               5.498718e+12  5.473738e+12  5.569102e+12  5.644659e+12   \n",
       "United Kingdom      2.403504e+12  2.450911e+12  2.479809e+12  2.533370e+12   \n",
       "Russian Federation  1.524917e+12  1.589943e+12  1.645876e+12  1.666934e+12   \n",
       "Canada              1.613406e+12  1.664087e+12  1.693133e+12  1.730688e+12   \n",
       "Germany             3.417298e+12  3.542371e+12  3.556724e+12  3.567317e+12   \n",
       "India               1.708459e+12  1.821872e+12  1.924235e+12  2.051982e+12   \n",
       "France              2.646995e+12  2.702032e+12  2.706968e+12  2.722567e+12   \n",
       "South Korea         1.094499e+12  1.134796e+12  1.160809e+12  1.194429e+12   \n",
       "Italy               2.125185e+12  2.137439e+12  2.077184e+12  2.040871e+12   \n",
       "Spain               1.431673e+12  1.417355e+12  1.380216e+12  1.357139e+12   \n",
       "Iran                4.677902e+11  4.853309e+11  4.532569e+11  4.445926e+11   \n",
       "Australia           1.142251e+12  1.169431e+12  1.211913e+12  1.241484e+12   \n",
       "Brazil              2.208872e+12  2.295245e+12  2.339209e+12  2.409740e+12   \n",
       "\n",
       "                            2014          2015  Rank  Documents  \\\n",
       "Country                                                           \n",
       "China               8.230121e+12  8.797999e+12     1     127050   \n",
       "United States       1.615662e+13  1.654857e+13     2      96661   \n",
       "Japan               5.642884e+12  5.669563e+12     3      30504   \n",
       "United Kingdom      2.605643e+12  2.666333e+12     4      20944   \n",
       "Russian Federation  1.678709e+12  1.616149e+12     5      18534   \n",
       "Canada              1.773486e+12  1.792609e+12     6      17899   \n",
       "Germany             3.624386e+12  3.685556e+12     7      17027   \n",
       "India               2.200617e+12  2.367206e+12     8      15005   \n",
       "France              2.729632e+12  2.761185e+12     9      13153   \n",
       "South Korea         1.234340e+12  1.266580e+12    10      11983   \n",
       "Italy               2.033868e+12  2.049316e+12    11      10964   \n",
       "Spain               1.375605e+12  1.419821e+12    12       9428   \n",
       "Iran                4.639027e+11           NaN    13       8896   \n",
       "Australia           1.272520e+12  1.301251e+12    14       8831   \n",
       "Brazil              2.412231e+12  2.319423e+12    15       8668   \n",
       "\n",
       "                    Citable documents  Citations  Self-citations  \\\n",
       "Country                                                            \n",
       "China                          126767     597237          411683   \n",
       "United States                   94747     792274          265436   \n",
       "Japan                           30287     223024           61554   \n",
       "United Kingdom                  20357     206091           37874   \n",
       "Russian Federation              18301      34266           12422   \n",
       "Canada                          17620     215003           40930   \n",
       "Germany                         16831     140566           27426   \n",
       "India                           14841     128763           37209   \n",
       "France                          12973     130632           28601   \n",
       "South Korea                     11923     114675           22595   \n",
       "Italy                           10794     111850           26661   \n",
       "Spain                            9330     123336           23964   \n",
       "Iran                             8819      57470           19125   \n",
       "Australia                        8725      90765           15606   \n",
       "Brazil                           8596      60702           14396   \n",
       "\n",
       "                    Citations per document  H index  \n",
       "Country                                              \n",
       "China                                 4.70      138  \n",
       "United States                         8.20      230  \n",
       "Japan                                 7.31      134  \n",
       "United Kingdom                        9.84      139  \n",
       "Russian Federation                    1.85       57  \n",
       "Canada                               12.01      149  \n",
       "Germany                               8.26      126  \n",
       "India                                 8.58      115  \n",
       "France                                9.93      114  \n",
       "South Korea                           9.57      104  \n",
       "Italy                                10.20      106  \n",
       "Spain                                13.08      115  \n",
       "Iran                                  6.46       72  \n",
       "Australia                            10.28      107  \n",
       "Brazil                                7.00       86  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_one():\n",
    "    energy = pd.read_excel('Energy Indicators.xls', skiprows=[i for i in range(17)], skipfooter=32)\n",
    "    energy = (energy.drop(energy.columns[[0, 1]], axis=1))\n",
    "    energy.columns = ['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']\n",
    "    energy['Energy Supply'] = energy['Energy Supply'].map(lambda a: a * 1000000. if isinstance(a, nb) else np.NaN)\n",
    "    energy['Country'] = energy['Country'].map(lambda a: str(a)[:str(a).find('(')-1] if str(a).find('(') != -1 \n",
    "                                              else a)\n",
    "    energy['Country'] = energy['Country'].map(lambda a: re.sub(\"\\d\", \"\", a) if any(map(str.isdigit, str(a)))\n",
    "                                             else a)\n",
    "    energy['Country'] = energy['Country'].map(lambda a: \"South Korea\" if str(a) == \"Republic of Korea\"\n",
    "                                              else \"United States\" if str(a) == \"United States of America\" \n",
    "                                              else \"United Kingdom\" if str(a) == \"United Kingdom of Great Britain and Northern Ireland\"\n",
    "                                              else \"Hong Kong\" if str(a) == \"China, Hong Kong Special Administrative Region\"\n",
    "                                              else a)\n",
    "    #energy.rename(columns={\"Country\": \"Country Name\"}, inplace=True)\n",
    "    energy.set_index('Country', inplace=True)\n",
    "\n",
    "    GDP = pd.read_csv(\"world_bank.csv\", skiprows=[0, 1, 2, 3])\n",
    "    GDP['Country Name'] = GDP['Country Name'].map(lambda a: \"South Korea\" if str(a) == \"Korea, Rep.\"\n",
    "                                                  else \"Iran\" if str(a) == \"Iran, Islamic Rep.\"\n",
    "                                                  else \"Hong Kong\" if str(a) == \"Hong Kong SAR, China\"\n",
    "                                                  else a)\n",
    "\n",
    "    ScimEn = pd.read_excel(\"scimagojr-3.xlsx\").head(15)\n",
    "    ScimEn.set_index('Country', inplace=True)\n",
    "    #ScimEn.rename(columns={\"Country\": \"Country Name\"}, inplace=True)\n",
    "\n",
    "    columns_to_keep_GDP = ['Country Name', '2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015']\n",
    "    GDP = GDP[columns_to_keep_GDP]\n",
    "    GDP.rename(columns={\"Country Name\": \"Country\"}, inplace=True)\n",
    "    GDP.set_index('Country', inplace=True)\n",
    "\n",
    "    result = pd.merge(energy, GDP, how='inner', left_index=True, right_index=True)\n",
    "    result = pd.merge(result, ScimEn, how='inner', left_index=True, right_index=True)\n",
    "    #result.set_index('Country', inplace=True)\n",
    "    return result\n",
    "answer_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2 (6.6%)\n",
    "The previous question joined three datasets then reduced this to just the top 15 entries. When you joined the datasets, but before you reduced this to the top 15 items, how many entries did you lose?\n",
    "\n",
    "*This function should return a single number.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg width=\"800\" height=\"300\">\n",
       "  <circle cx=\"150\" cy=\"180\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"blue\" />\n",
       "  <circle cx=\"200\" cy=\"100\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"red\" />\n",
       "  <circle cx=\"100\" cy=\"100\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"green\" />\n",
       "  <line x1=\"150\" y1=\"125\" x2=\"300\" y2=\"150\" stroke=\"black\" stroke-width=\"2\" fill=\"black\" stroke-dasharray=\"5,3\"/>\n",
       "  <text  x=\"300\" y=\"165\" font-family=\"Verdana\" font-size=\"35\">Everything but this!</text>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<svg width=\"800\" height=\"300\">\n",
    "  <circle cx=\"150\" cy=\"180\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"blue\" />\n",
    "  <circle cx=\"200\" cy=\"100\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"red\" />\n",
    "  <circle cx=\"100\" cy=\"100\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"green\" />\n",
    "  <line x1=\"150\" y1=\"125\" x2=\"300\" y2=\"150\" stroke=\"black\" stroke-width=\"2\" fill=\"black\" stroke-dasharray=\"5,3\"/>\n",
    "  <text  x=\"300\" y=\"165\" font-family=\"Verdana\" font-size=\"35\">Everything but this!</text>\n",
    "</svg>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "umich_part_id": "010",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "176"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_two():\n",
    "    n_of_entries = 15\n",
    "    ScimEn = pd.read_excel(\"scimagojr-3.xlsx\")\n",
    "    return ScimEn.shape[0] - n_of_entries\n",
    "answer_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer the following questions in the context of only the top 15 countries by Scimagojr Rank (aka the DataFrame returned by `answer_one()`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3 (6.6%)\n",
    "What is the average GDP over the last 10 years for each country? (exclude missing values from this calculation.)\n",
    "\n",
    "*This function should return a Series named `avgGDP` with 15 countries and their average GDP sorted in descending order.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true,
    "umich_part_id": "011",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country\n",
       "United States         1.536434e+13\n",
       "China                 6.348609e+12\n",
       "Japan                 5.542208e+12\n",
       "Germany               3.493025e+12\n",
       "France                2.681725e+12\n",
       "United Kingdom        2.487907e+12\n",
       "Brazil                2.189794e+12\n",
       "Italy                 2.120175e+12\n",
       "India                 1.769297e+12\n",
       "Canada                1.660647e+12\n",
       "Russian Federation    1.565459e+12\n",
       "Spain                 1.418078e+12\n",
       "Australia             1.164043e+12\n",
       "South Korea           1.106715e+12\n",
       "Iran                  4.441558e+11\n",
       "dtype: float64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_three():\n",
    "    Top15 = answer_one()\n",
    "    columns_to_use = ['2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015']\n",
    "    df = Top15[columns_to_use]\n",
    "    avgGDP = df.mean(axis = 1, skipna = True).sort_values(ascending=False)\n",
    "    return avgGDP\n",
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4 (6.6%)\n",
    "By how much had the GDP changed over the 10 year span for the country with the 6th largest average GDP?\n",
    "\n",
    "*This function should return a single number.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true,
    "umich_part_id": "012",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "246702696075.3999"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_four():\n",
    "    Top15 = answer_one()\n",
    "    columns_to_use = ['2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015']\n",
    "    df = Top15[columns_to_use]\n",
    "    df['Average'] = df.mean(axis = 1, skipna = True)\n",
    "    df.sort_values(by=['Average'], ascending=False, inplace=True)\n",
    "    sixth = df.iloc[5]\n",
    "    diff = sixth['2015'] - sixth['2006']\n",
    "    return diff\n",
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5 (6.6%)\n",
    "What is the mean `Energy Supply per Capita`?\n",
    "\n",
    "*This function should return a single number.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "umich_part_id": "013",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "157.59999999999999"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_five():\n",
    "    Top15 = answer_one()\n",
    "    df = Top15.reset_index()\n",
    "    df['Energy Supply per Capita'] = df['Energy Supply per Capita'].map(lambda a: a if str(a).isdigit() else np.NaN)\n",
    "    return df['Energy Supply per Capita'].mean()\n",
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6 (6.6%)\n",
    "What country has the maximum % Renewable and what is the percentage?\n",
    "\n",
    "*This function should return a tuple with the name of the country and the percentage.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "umich_part_id": "014",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Brazil', 69.648030000000006)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_six():\n",
    "    Top15 = answer_one()\n",
    "    df = Top15['% Renewable']\n",
    "    df = df.map(lambda a: np.NaN if not isinstance(a, nb) else a).sort_values(ascending=False)\n",
    "    return (df.index[0], df[0])\n",
    "answer_six()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7 (6.6%)\n",
    "Create a new column that is the ratio of Self-Citations to Total Citations. \n",
    "What is the maximum value for this new column, and what country has the highest ratio?\n",
    "\n",
    "*This function should return a tuple with the name of the country and the ratio.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "umich_part_id": "015",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('China', 0.68931261793894216)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_seven():\n",
    "    Top15 = answer_one()\n",
    "    Top15['Ratio'] = (Top15['Self-citations'] / Top15['Citations'])\n",
    "    needed_column = Top15['Ratio']\n",
    "    needed_column.sort_values(ascending=False)\n",
    "    return (needed_column.index[0], needed_column[0])\n",
    "answer_seven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 8 (6.6%)\n",
    "\n",
    "Create a column that estimates the population using Energy Supply and Energy Supply per capita. \n",
    "What is the third most populous country according to this estimate?\n",
    "\n",
    "*This function should return a single string value.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "umich_part_id": "016",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'United States'"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eight():\n",
    "    Top15 = answer_one()\n",
    "    Top15['Population'] = Top15['Energy Supply'] / Top15['Energy Supply per Capita']\n",
    "    Top15.sort_values(by=\"Population\", ascending=False, inplace=True)\n",
    "    return Top15.index[2]\n",
    "answer_eight()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9 (6.6%)\n",
    "Create a column that estimates the number of citable documents per person. \n",
    "What is the correlation between the number of citable documents per capita and the energy supply per capita? Use the `.corr()` method, (Pearson's correlation).\n",
    "\n",
    "*This function should return a single number.*\n",
    "\n",
    "*(Optional: Use the built-in function `plot9()` to visualize the relationship between Energy Supply per Capita vs. Citable docs per Capita)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "umich_part_id": "017",
    "umich_partlist_id": "003"
   },
   "outputs": [],
   "source": [
    "def answer_nine():\n",
    "    Top15 = answer_one()\n",
    "    return \"ANSWER\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot9():\n",
    "    import matplotlib as plt\n",
    "    %matplotlib inline\n",
    "    \n",
    "    Top15 = answer_one()\n",
    "    Top15['PopEst'] = Top15['Energy Supply'] / Top15['Energy Supply per Capita']\n",
    "    Top15['Citable docs per Capita'] = Top15['Citable documents'] / Top15['PopEst']\n",
    "    Top15.plot(x='Citable docs per Capita', y='Energy Supply per Capita', kind='scatter', xlim=[0, 0.0006])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#plot9() # Be sure to comment out plot9() before submitting the assignment!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10 (6.6%)\n",
    "Create a new column with a 1 if the country's % Renewable value is at or above the median for all countries in the top 15, and a 0 if the country's % Renewable value is below the median.\n",
    "\n",
    "*This function should return a series named `HighRenew` whose index is the country name sorted in ascending order of rank.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "umich_part_id": "018",
    "umich_partlist_id": "003"
   },
   "outputs": [],
   "source": [
    "def answer_ten():\n",
    "    Top15 = answer_one()\n",
    "    return \"ANSWER\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 11 (6.6%)\n",
    "Use the following dictionary to group the Countries by Continent, then create a dateframe that displays the sample size (the number of countries in each continent bin), and the sum, mean, and std deviation for the estimated population of each country.\n",
    "\n",
    "```python\n",
    "ContinentDict  = {'China':'Asia', \n",
    "                  'United States':'North America', \n",
    "                  'Japan':'Asia', \n",
    "                  'United Kingdom':'Europe', \n",
    "                  'Russian Federation':'Europe', \n",
    "                  'Canada':'North America', \n",
    "                  'Germany':'Europe', \n",
    "                  'India':'Asia',\n",
    "                  'France':'Europe', \n",
    "                  'South Korea':'Asia', \n",
    "                  'Italy':'Europe', \n",
    "                  'Spain':'Europe', \n",
    "                  'Iran':'Asia',\n",
    "                  'Australia':'Australia', \n",
    "                  'Brazil':'South America'}\n",
    "```\n",
    "\n",
    "*This function should return a DataFrame with index named Continent `['Asia', 'Australia', 'Europe', 'North America', 'South America']` and columns `['size', 'sum', 'mean', 'std']`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "umich_part_id": "019",
    "umich_partlist_id": "003"
   },
   "outputs": [],
   "source": [
    "def answer_eleven():\n",
    "    Top15 = answer_one()\n",
    "    return \"ANSWER\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 12 (6.6%)\n",
    "Cut % Renewable into 5 bins. Group Top15 by the Continent, as well as these new % Renewable bins. How many countries are in each of these groups?\n",
    "\n",
    "*This function should return a __Series__ with a MultiIndex of `Continent`, then the bins for `% Renewable`. Do not include groups with no countries.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def answer_twelve():\n",
    "    Top15 = answer_one()\n",
    "    return \"ANSWER\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 13 (6.6%)\n",
    "Convert the Population Estimate series to a string with thousands separator (using commas). Do not round the results.\n",
    "\n",
    "e.g. 317615384.61538464 -> 317,615,384.61538464\n",
    "\n",
    "*This function should return a Series `PopEst` whose index is the country name and whose values are the population estimate string.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true,
    "umich_part_id": "020",
    "umich_partlist_id": "003"
   },
   "outputs": [],
   "source": [
    "def answer_thirteen():\n",
    "    Top15 = answer_one()\n",
    "    return \"ANSWER\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional\n",
    "\n",
    "Use the built in function `plot_optional()` to see an example visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def plot_optional():\n",
    "    import matplotlib as plt\n",
    "    %matplotlib inline\n",
    "    Top15 = answer_one()\n",
    "    ax = Top15.plot(x='Rank', y='% Renewable', kind='scatter', \n",
    "                    c=['#e41a1c','#377eb8','#e41a1c','#4daf4a','#4daf4a','#377eb8','#4daf4a','#e41a1c',\n",
    "                       '#4daf4a','#e41a1c','#4daf4a','#4daf4a','#e41a1c','#dede00','#ff7f00'], \n",
    "                    xticks=range(1,16), s=6*Top15['2014']/10**10, alpha=.75, figsize=[16,6]);\n",
    "\n",
    "    for i, txt in enumerate(Top15.index):\n",
    "        ax.annotate(txt, [Top15['Rank'][i], Top15['% Renewable'][i]], ha='center')\n",
    "\n",
    "    print(\"This is an example of a visualization that can be created to help understand the data. \\\n",
    "This is a bubble chart showing % Renewable vs. Rank. The size of the bubble corresponds to the countries' \\\n",
    "2014 GDP, and the color corresponds to the continent.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#plot_optional() # Be sure to comment out plot_optional() before submitting the assignment!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "coursera": {
   "course_slug": "python-data-analysis",
   "graded_item_id": "zAr06",
   "launcher_item_id": "KSSjT",
   "part_id": "SL3fU"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  },
  "umich": {
   "id": "Assignment 3",
   "version": "1.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}