Module_0_Basic_Python_Programming/03_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas\n",
    "\n",
    "Pandas is an easy-to-use, fast, flexible, and powerful open-source Python library for working with “relational” or “labeled” data. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python by offering data structures and operations for manipulating tables. And, like Numpy, a Pandas implementation is faster than a default Python ones, and it is a third-party module, which means it is not part of Python's suite of built-in libraries. \n",
    "\n",
    "Pandas is a very big topic and it has so many features that cannot be tacked in this module. You can do further studying by reading the official documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.html.\n",
    "\n",
    "**Import**\n",
    "\n",
    "Again, we have to import the library first, similar to numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The alias `pd` is a common namin convention for pandas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that Jupyter has the following features: tab-completion(<TAB>) and `?`. You may use this to explore and access the documentation for anything that you are looking for.\n",
    "\n",
    "```\n",
    "Example:\n",
    "\n",
    "ipython\n",
    "In [1]: pd.<TAB>\n",
    "or\n",
    "\n",
    "ipython\n",
    "In [2]: pd.__version__?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data Strucures**\n",
    "\n",
    "Pandas has 2 data structures: `Series` and `Dataframe`. These 2 give us many features that make it easy to do data analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Series\n",
    "\n",
    "Pandas Series is a one-dimensional array-like object that has index and value just like NumPy and is capable of holding any data type. \n",
    "\n",
    "**Creating Series**\n",
    "\n",
    "You can input a list, numpy array, and dictionary to create a Series. Here are some examples of showcasing those."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a   -0.528704\n",
      "b    0.517890\n",
      "c   -1.201350\n",
      "d   -1.080025\n",
      "e    2.044870\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "s = pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.series.Series'>\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(type(s))\n",
    "print(s.name)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access the values by indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.5287041215347653"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s['a']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you don't pass an argument for index, it uses 0..n by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:  test_data\n",
      "Data:\n",
      " 0    8\n",
      "1    7\n",
      "2    6\n",
      "3    5\n",
      "Name: test_data, dtype: int64\n",
      "Type of Object:  <class 'pandas.core.series.Series'>\n",
      "Type of elements: <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "s = pd.Series([8,7,6,5], name='test_data')\n",
    "print('Name: ',s.name)\n",
    "print('Data:\\n',s)\n",
    "print('Type of Object: ',type(s))\n",
    "print('Type of elements:',type(s.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also create a Series from a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Library1         Numpy\n",
      "Library2        Pandas\n",
      "Library3    Matplotlib\n",
      "dtype: object\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "libs_dict = {'Library1': 'Numpy', \n",
    "               'Library2': 'Pandas', \n",
    "               'Library3': 'Matplotlib'}\n",
    "s = pd.Series(libs_dict)\n",
    "print(s)\n",
    "print(type(s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Pandas'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s['Library2']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, when creating a Pandas Series, only the data (first argument) is mandatory here. The rest are optional; you can opt not to input the name or index.\n",
    "\n",
    "**Editing Series**\n",
    "\n",
    "Editing a Series is very similar to how you do it with `dict`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Library1         Numpy\n",
      "Library2        Pandas\n",
      "Library3    Matplotlib\n",
      "dtype: object\n",
      "Library1         Numpy\n",
      "Library2    Pandas 2.0\n",
      "Library3    Matplotlib\n",
      "Library4     GeoPandas\n",
      "dtype: object\n",
      "Library1         Numpy\n",
      "Library2    Pandas 2.0\n",
      "Library3    Matplotlib\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "libs_dict = {'Library1': 'Numpy', 'Library2': 'Pandas', 'Library3': 'Matplotlib'}\n",
    "s = pd.Series(libs_dict)\n",
    "print(s)\n",
    "\n",
    "s['Library2'] = 'Pandas 2.0'\n",
    "s['Library4'] = 'GeoPandas'\n",
    "print(s)\n",
    "\n",
    "s.pop('Library4')\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Dataframe\n",
    "\n",
    "Dataframe is like spreadsheet or a SQL table. It is basically a 2-dimensional labelled data structure with columns of potentially different data types. To put it simply, `DataFrame` is a multi-column `Series` object. It is generally the most commonly used pandas object, and like `Series`, `DataFrame` accepts many different kinds of input:\n",
    "\n",
    " - Dict of 1D ndarrays, lists, dicts, or Series\n",
    " - 2-D numpy.ndarray\n",
    " - Structured or record ndarray\n",
    " - A `Series`\n",
    " - Another `DataFrame`\n",
    "  \n",
    "Here are some examples of creating Dataframes using different data types and structures as inputs.\n",
    "\n",
    "**Creating Dataframe using dict**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe:\n",
      "    one  two\n",
      "a  1.0  1.0\n",
      "b  2.0  2.0\n",
      "c  3.0  3.0\n",
      "d  NaN  4.0\n",
      "Type of Object: <class 'pandas.core.frame.DataFrame'>\n",
      "Type of elements: <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "data = {'one' : pd.Series([1., 2., 3.], index=['a', 'b', 'c']),\n",
    "    'two' : pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "print('Dataframe:\\n',df)\n",
    "print('Type of Object:',type(df))\n",
    "print('Type of elements:',type(df.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like Series, you can access index, values and also columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index:  Index(['a', 'b', 'c', 'd'], dtype='object')\n",
      "Columns:  Index(['one', 'two'], dtype='object')\n",
      "Values of Column one:  [ 1.  2.  3. nan]\n",
      "Values of Column two:  [1. 2. 3. 4.]\n"
     ]
    }
   ],
   "source": [
    "print('Index: ',df.index)\n",
    "print('Columns: ',df.columns)\n",
    "print('Values of Column one: ',df['one'].values)\n",
    "print('Values of Column two: ',df['two'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating Dataframe using list of dict**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe: \n",
      "      a  b  c    d\n",
      "one  1  2  3  NaN\n",
      "two  2  2  3  4.0\n",
      "Transposed Dataframe: \n",
      "    one  two\n",
      "a  1.0  2.0\n",
      "b  2.0  2.0\n",
      "c  3.0  3.0\n",
      "d  NaN  4.0\n"
     ]
    }
   ],
   "source": [
    "df2 = pd.DataFrame([{'a': 1, 'b': 2, 'c':3, 'd':None}, \n",
    "                    {'a': 2, 'b': 2, 'c': 3, 'd': 4}],\n",
    "                   index=['one', 'two'])\n",
    "print('Dataframe: \\n',df2)\n",
    "\n",
    "# Ofcourse you can also transpose the result:\n",
    "print('Transposed Dataframe: \\n',df2.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Editing DataFrame**\n",
    "\n",
    "Assigning a column that doesn’t exist will create a new column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added third column: \n",
      "    one  two three\n",
      "a  1.0  1.0  None\n",
      "b  2.0  2.0  None\n",
      "c  3.0  3.0  None\n",
      "d  NaN  4.0  None\n",
      "\n",
      "Deleted third column: \n",
      "    one  two\n",
      "a  1.0  1.0\n",
      "b  2.0  2.0\n",
      "c  3.0  3.0\n",
      "d  NaN  4.0\n",
      "\n",
      "Edited a value: \n",
      "       one  two\n",
      "a  9000.0  1.0\n",
      "b     2.0  2.0\n",
      "c     3.0  3.0\n",
      "d     NaN  4.0\n"
     ]
    }
   ],
   "source": [
    "df = pd.DataFrame(data)\n",
    "\n",
    "df['three'] = None\n",
    "print('Added third column: \\n',df)\n",
    "\n",
    "# The del keyword can be used delete columns:\n",
    "del df['three']\n",
    "print('\\nDeleted third column: \\n',df)\n",
    "# You can also use df.drop(). We shall see that later\n",
    "\n",
    "df.loc['a','one'] = 9000\n",
    "print('\\nEdited a value: \\n',df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Reindexing\n",
    "\n",
    "In Pandas, we can do reindexing, which means to create a new object with the data conformed to a new index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>one</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   one  two\n",
       "a  1.0  1.0\n",
       "b  2.0  2.0\n",
       "c  3.0  3.0\n",
       "d  NaN  4.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {'one' : pd.Series([1., 2., 3.], index=['a', 'b', 'c']),\n",
    "    'two' : pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}\n",
    "df = pd.DataFrame(data)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>one</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   one  two\n",
       "d  NaN  4.0\n",
       "c  3.0  3.0\n",
       "b  2.0  2.0\n",
       "a  1.0  1.0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reindex in descending order.\n",
    "df.reindex(['d','c','b','a'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you reindex with more number of rows than in the dataframe, it will return the dataframe with new row whose values are `NaN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>one</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   one  two\n",
       "a  1.0  1.0\n",
       "b  2.0  2.0\n",
       "c  3.0  3.0\n",
       "d  NaN  4.0\n",
       "e  NaN  NaN"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.reindex(['a','b','c','d','e'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reindexing is also useful when you want to introduce any missing values. For example in our case, look at column `one` and row `d`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>one</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   one  two\n",
       "a  1.0  1.0\n",
       "b  2.0  2.0\n",
       "c  3.0  3.0\n",
       "d  NaN  4.0\n",
       "e  0.0  0.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.reindex(['a','b','c','d','e'], fill_value=0)\n",
    "# Guess why the df['one']['d'] was not filled with 0 ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Dropping entries\n",
    "\n",
    "We can drop the rows/entries by using the `drop` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>one</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   one  two\n",
       "a  1.0  1.0\n",
       "b  2.0  2.0\n",
       "c  3.0  3.0\n",
       "d  NaN  4.0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {'one' : pd.Series([1., 2., 3.], index=['a', 'b', 'c']),\n",
    "    'two' : pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}\n",
    "df = pd.DataFrame(data)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>one</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   one  two\n",
       "b  2.0  2.0\n",
       "d  NaN  4.0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.drop(['c', 'a'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Indexing, Filtering, and Sorting \n",
    "\n",
    "**Indexing**\n",
    "\n",
    "Series indexing works similarly to NumPy array indexing, except you can use the Series’s index values instead of only integers.\n",
    "\n",
    "To slice and select only column one for rows 0 and 4 use the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe: \n",
      "    one  two\n",
      "a  1.0  1.0\n",
      "b  2.0  2.0\n",
      "c  3.0  3.0\n",
      "d  NaN  4.0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "a    1.0\n",
       "d    NaN\n",
       "Name: one, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Dataframe: \\n\",df)\n",
    "# Slicing and selecting only column `one` for row 0 and row 4\n",
    "df['one'][['a', 'd']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b    2.0\n",
       "c    3.0\n",
       "d    NaN\n",
       "Name: one, dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Slicing df from row b to row 4 for column `one`\n",
    "df['one']['b':'d']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above cell, you will notice that slicing with labels behaves differently than normal Python slicing in that the endpoint is inclusive.\n",
    "\n",
    "For DataFrame label-indexing on the rows, there is a special indexing field loc, which enables us to select a subset of the rows and columns from a `DataFrame` with `numpy`-like notation plus axis labels. It is a less verbose way to do the reindexing, but we typically use this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>one</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   one\n",
       "a  1.0\n",
       "c  3.0"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[['a','c'],['one']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Filtering**\n",
    "\n",
    "We can also filter by having a condition inside `loc`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>one</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   one  two\n",
       "b  2.0  2.0\n",
       "c  3.0  3.0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[df.one > 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many ways to select and rearrange the data contained in a pandas object. Some indexing options can be seen in below table:\n",
    "\n",
    "|Indexing Type| Description|\n",
    "|:---|:---|\n",
    "|df[val] | Select single column or sequence of columns from the DataFrame. Special case con- veniences: boolean array (filter rows), slice (slice rows), or boolean DataFrame (set values based on some criterion).|\n",
    "|df.ix[val] | Selects single row of subset of rows from the DataFrame.|\n",
    "|df.ix[:, val] | Selects single column of subset of columns.|\n",
    "|df.ix[val1, val2] | Select both rows and columns.|\n",
    "|reindex method | Conform one or more axes to new indexes.|\n",
    "|xs method | Select single row or column as a Series by label.|\n",
    "|icol, irowmethods | Select single column or row, respectively, as a Series by integer location.|\n",
    "|get_value, set_value methods | Select single value by row and column label.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Sorting**\n",
    "\n",
    "We can sort items by doing the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original Data: \n",
      " g    4\n",
      "c    7\n",
      "a    6\n",
      "b    7\n",
      "e    8\n",
      "d    3\n",
      "f    9\n",
      "dtype: int64\n",
      "\n",
      "Sorted by Index: \n",
      " a    6\n",
      "b    7\n",
      "c    7\n",
      "d    3\n",
      "e    8\n",
      "f    9\n",
      "g    4\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "dt = pd.Series(np.random.randint(3, 10, size=7), \n",
    "               index=['g','c','a','b','e','d','f'])\n",
    "print('Original Data: \\n', dt, end=\"\\n\\n\")\n",
    "print('Sorted by Index: \\n',dt.sort_index())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Data alignment and arithmetic\n",
    "\n",
    "Data alignment between DataFrame objects automatically align on both the columns and the index (row labels). The resulting object will have the union of the column and row labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df1:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.075197</td>\n",
       "      <td>1.071788</td>\n",
       "      <td>-0.346722</td>\n",
       "      <td>0.200615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.344002</td>\n",
       "      <td>-0.245280</td>\n",
       "      <td>-1.422829</td>\n",
       "      <td>0.108585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.536841</td>\n",
       "      <td>0.015907</td>\n",
       "      <td>-0.734943</td>\n",
       "      <td>0.749065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.988480</td>\n",
       "      <td>0.489212</td>\n",
       "      <td>-1.426875</td>\n",
       "      <td>0.164377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.468510</td>\n",
       "      <td>-0.979854</td>\n",
       "      <td>0.293018</td>\n",
       "      <td>0.944181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.043697</td>\n",
       "      <td>-0.211873</td>\n",
       "      <td>-1.554627</td>\n",
       "      <td>-0.015371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.585974</td>\n",
       "      <td>1.262029</td>\n",
       "      <td>0.403454</td>\n",
       "      <td>-0.657079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.309760</td>\n",
       "      <td>-0.230141</td>\n",
       "      <td>1.626267</td>\n",
       "      <td>1.151040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-0.869131</td>\n",
       "      <td>0.738002</td>\n",
       "      <td>1.211384</td>\n",
       "      <td>2.257951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.034295</td>\n",
       "      <td>-1.547727</td>\n",
       "      <td>1.515287</td>\n",
       "      <td>0.304565</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C         D\n",
       "0 -0.075197  1.071788 -0.346722  0.200615\n",
       "1  1.344002 -0.245280 -1.422829  0.108585\n",
       "2  0.536841  0.015907 -0.734943  0.749065\n",
       "3  0.988480  0.489212 -1.426875  0.164377\n",
       "4 -0.468510 -0.979854  0.293018  0.944181\n",
       "5  0.043697 -0.211873 -1.554627 -0.015371\n",
       "6  0.585974  1.262029  0.403454 -0.657079\n",
       "7  0.309760 -0.230141  1.626267  1.151040\n",
       "8 -0.869131  0.738002  1.211384  2.257951\n",
       "9  1.034295 -1.547727  1.515287  0.304565"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df2:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1.151598</td>\n",
       "      <td>-1.719839</td>\n",
       "      <td>-1.444585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.252279</td>\n",
       "      <td>0.069754</td>\n",
       "      <td>-1.196836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.378880</td>\n",
       "      <td>0.461332</td>\n",
       "      <td>0.059675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.824091</td>\n",
       "      <td>-0.427750</td>\n",
       "      <td>0.985102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.368444</td>\n",
       "      <td>-0.342445</td>\n",
       "      <td>-0.537114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.091544</td>\n",
       "      <td>-0.065743</td>\n",
       "      <td>-0.066165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.031548</td>\n",
       "      <td>-0.109830</td>\n",
       "      <td>-0.370547</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C\n",
       "0 -1.151598 -1.719839 -1.444585\n",
       "1  1.252279  0.069754 -1.196836\n",
       "2  1.378880  0.461332  0.059675\n",
       "3 -0.824091 -0.427750  0.985102\n",
       "4 -1.368444 -0.342445 -0.537114\n",
       "5  1.091544 -0.065743 -0.066165\n",
       "6  1.031548 -0.109830 -0.370547"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sum:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1.226796</td>\n",
       "      <td>-0.648051</td>\n",
       "      <td>-1.791307</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.596282</td>\n",
       "      <td>-0.175525</td>\n",
       "      <td>-2.619664</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.915721</td>\n",
       "      <td>0.477239</td>\n",
       "      <td>-0.675268</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.164389</td>\n",
       "      <td>0.061461</td>\n",
       "      <td>-0.441773</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.836953</td>\n",
       "      <td>-1.322298</td>\n",
       "      <td>-0.244097</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.135241</td>\n",
       "      <td>-0.277616</td>\n",
       "      <td>-1.620792</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.617523</td>\n",
       "      <td>1.152200</td>\n",
       "      <td>0.032908</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C   D\n",
       "0 -1.226796 -0.648051 -1.791307 NaN\n",
       "1  2.596282 -0.175525 -2.619664 NaN\n",
       "2  1.915721  0.477239 -0.675268 NaN\n",
       "3  0.164389  0.061461 -0.441773 NaN\n",
       "4 -1.836953 -1.322298 -0.244097 NaN\n",
       "5  1.135241 -0.277616 -1.620792 NaN\n",
       "6  1.617523  1.152200  0.032908 NaN\n",
       "7       NaN       NaN       NaN NaN\n",
       "8       NaN       NaN       NaN NaN\n",
       "9       NaN       NaN       NaN NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df1 = pd.DataFrame(np.random.randn(10, 4), columns=['A', 'B', 'C', 'D'])\n",
    "df2 = pd.DataFrame(np.random.randn(7, 3), columns=['A', 'B', 'C'])\n",
    "\n",
    "print('df1:')\n",
    "display(df1)\n",
    "print('df2:')\n",
    "display(df2)\n",
    "print('Sum:')\n",
    "display(df1.add(df2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in arithmetic operations between differently-indexed objects, you might want to fill with a special value, like 0, when an axis label is found in one object but not the other:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1.226796</td>\n",
       "      <td>-0.648051</td>\n",
       "      <td>-1.791307</td>\n",
       "      <td>0.200615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.596282</td>\n",
       "      <td>-0.175525</td>\n",
       "      <td>-2.619664</td>\n",
       "      <td>0.108585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.915721</td>\n",
       "      <td>0.477239</td>\n",
       "      <td>-0.675268</td>\n",
       "      <td>0.749065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.164389</td>\n",
       "      <td>0.061461</td>\n",
       "      <td>-0.441773</td>\n",
       "      <td>0.164377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.836953</td>\n",
       "      <td>-1.322298</td>\n",
       "      <td>-0.244097</td>\n",
       "      <td>0.944181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.135241</td>\n",
       "      <td>-0.277616</td>\n",
       "      <td>-1.620792</td>\n",
       "      <td>-0.015371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.617523</td>\n",
       "      <td>1.152200</td>\n",
       "      <td>0.032908</td>\n",
       "      <td>-0.657079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.309760</td>\n",
       "      <td>-0.230141</td>\n",
       "      <td>1.626267</td>\n",
       "      <td>1.151040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-0.869131</td>\n",
       "      <td>0.738002</td>\n",
       "      <td>1.211384</td>\n",
       "      <td>2.257951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.034295</td>\n",
       "      <td>-1.547727</td>\n",
       "      <td>1.515287</td>\n",
       "      <td>0.304565</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C         D\n",
       "0 -1.226796 -0.648051 -1.791307  0.200615\n",
       "1  2.596282 -0.175525 -2.619664  0.108585\n",
       "2  1.915721  0.477239 -0.675268  0.749065\n",
       "3  0.164389  0.061461 -0.441773  0.164377\n",
       "4 -1.836953 -1.322298 -0.244097  0.944181\n",
       "5  1.135241 -0.277616 -1.620792 -0.015371\n",
       "6  1.617523  1.152200  0.032908 -0.657079\n",
       "7  0.309760 -0.230141  1.626267  1.151040\n",
       "8 -0.869131  0.738002  1.211384  2.257951\n",
       "9  1.034295 -1.547727  1.515287  0.304565"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.add(df2, fill_value=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, you can perform other arithmetic operations such as subtracion, multiplication and division.\n",
    "\n",
    "When doing an operation between DataFrame and Series, the default behavior is to align the Series index on the DataFrame columns, thus broadcasting (just like in numpy) row-wise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.075197</td>\n",
       "      <td>1.071788</td>\n",
       "      <td>-0.346722</td>\n",
       "      <td>0.200615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.344002</td>\n",
       "      <td>-0.245280</td>\n",
       "      <td>-1.422829</td>\n",
       "      <td>0.108585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.536841</td>\n",
       "      <td>0.015907</td>\n",
       "      <td>-0.734943</td>\n",
       "      <td>0.749065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.988480</td>\n",
       "      <td>0.489212</td>\n",
       "      <td>-1.426875</td>\n",
       "      <td>0.164377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.468510</td>\n",
       "      <td>-0.979854</td>\n",
       "      <td>0.293018</td>\n",
       "      <td>0.944181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.043697</td>\n",
       "      <td>-0.211873</td>\n",
       "      <td>-1.554627</td>\n",
       "      <td>-0.015371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.585974</td>\n",
       "      <td>1.262029</td>\n",
       "      <td>0.403454</td>\n",
       "      <td>-0.657079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.309760</td>\n",
       "      <td>-0.230141</td>\n",
       "      <td>1.626267</td>\n",
       "      <td>1.151040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-0.869131</td>\n",
       "      <td>0.738002</td>\n",
       "      <td>1.211384</td>\n",
       "      <td>2.257951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.034295</td>\n",
       "      <td>-1.547727</td>\n",
       "      <td>1.515287</td>\n",
       "      <td>0.304565</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C         D\n",
       "0 -0.075197  1.071788 -0.346722  0.200615\n",
       "1  1.344002 -0.245280 -1.422829  0.108585\n",
       "2  0.536841  0.015907 -0.734943  0.749065\n",
       "3  0.988480  0.489212 -1.426875  0.164377\n",
       "4 -0.468510 -0.979854  0.293018  0.944181\n",
       "5  0.043697 -0.211873 -1.554627 -0.015371\n",
       "6  0.585974  1.262029  0.403454 -0.657079\n",
       "7  0.309760 -0.230141  1.626267  1.151040\n",
       "8 -0.869131  0.738002  1.211384  2.257951\n",
       "9  1.034295 -1.547727  1.515287  0.304565"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Operand (0th row):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "A   -0.075197\n",
       "B    1.071788\n",
       "C   -0.346722\n",
       "D    0.200615\n",
       "Name: 0, dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subtraction:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.419199</td>\n",
       "      <td>-1.317067</td>\n",
       "      <td>-1.076107</td>\n",
       "      <td>-0.092030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.612038</td>\n",
       "      <td>-1.055881</td>\n",
       "      <td>-0.388221</td>\n",
       "      <td>0.548449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.063678</td>\n",
       "      <td>-0.582576</td>\n",
       "      <td>-1.080153</td>\n",
       "      <td>-0.036238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.393312</td>\n",
       "      <td>-2.051642</td>\n",
       "      <td>0.639739</td>\n",
       "      <td>0.743566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.118894</td>\n",
       "      <td>-1.283661</td>\n",
       "      <td>-1.207905</td>\n",
       "      <td>-0.215986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.661172</td>\n",
       "      <td>0.190242</td>\n",
       "      <td>0.750176</td>\n",
       "      <td>-0.857694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.384958</td>\n",
       "      <td>-1.301929</td>\n",
       "      <td>1.972988</td>\n",
       "      <td>0.950425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-0.793934</td>\n",
       "      <td>-0.333786</td>\n",
       "      <td>1.558105</td>\n",
       "      <td>2.057335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.109492</td>\n",
       "      <td>-2.619515</td>\n",
       "      <td>1.862009</td>\n",
       "      <td>0.103950</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C         D\n",
       "0  0.000000  0.000000  0.000000  0.000000\n",
       "1  1.419199 -1.317067 -1.076107 -0.092030\n",
       "2  0.612038 -1.055881 -0.388221  0.548449\n",
       "3  1.063678 -0.582576 -1.080153 -0.036238\n",
       "4 -0.393312 -2.051642  0.639739  0.743566\n",
       "5  0.118894 -1.283661 -1.207905 -0.215986\n",
       "6  0.661172  0.190242  0.750176 -0.857694\n",
       "7  0.384958 -1.301929  1.972988  0.950425\n",
       "8 -0.793934 -0.333786  1.558105  2.057335\n",
       "9  1.109492 -2.619515  1.862009  0.103950"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Dataframe:\")\n",
    "display(df1)\n",
    "print(\"Operand (0th row):\")\n",
    "display(df1.loc[0])\n",
    "print('Subtraction:')\n",
    "display(df1.sub(df1.loc[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the special case of working with time series data, and the DataFrame index also contains dates, the broadcasting will be column-wise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017-06-01</th>\n",
       "      <td>-0.075197</td>\n",
       "      <td>1.071788</td>\n",
       "      <td>-0.346722</td>\n",
       "      <td>0.200615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-06-02</th>\n",
       "      <td>1.344002</td>\n",
       "      <td>-0.245280</td>\n",
       "      <td>-1.422829</td>\n",
       "      <td>0.108585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-06-03</th>\n",
       "      <td>0.536841</td>\n",
       "      <td>0.015907</td>\n",
       "      <td>-0.734943</td>\n",
       "      <td>0.749065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-06-04</th>\n",
       "      <td>0.988480</td>\n",
       "      <td>0.489212</td>\n",
       "      <td>-1.426875</td>\n",
       "      <td>0.164377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-06-05</th>\n",
       "      <td>-0.468510</td>\n",
       "      <td>-0.979854</td>\n",
       "      <td>0.293018</td>\n",
       "      <td>0.944181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-06-06</th>\n",
       "      <td>0.043697</td>\n",
       "      <td>-0.211873</td>\n",
       "      <td>-1.554627</td>\n",
       "      <td>-0.015371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-06-07</th>\n",
       "      <td>0.585974</td>\n",
       "      <td>1.262029</td>\n",
       "      <td>0.403454</td>\n",
       "      <td>-0.657079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-06-08</th>\n",
       "      <td>0.309760</td>\n",
       "      <td>-0.230141</td>\n",
       "      <td>1.626267</td>\n",
       "      <td>1.151040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-06-09</th>\n",
       "      <td>-0.869131</td>\n",
       "      <td>0.738002</td>\n",
       "      <td>1.211384</td>\n",
       "      <td>2.257951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-06-10</th>\n",
       "      <td>1.034295</td>\n",
       "      <td>-1.547727</td>\n",
       "      <td>1.515287</td>\n",
       "      <td>0.304565</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "2017-06-01 -0.075197  1.071788 -0.346722  0.200615\n",
       "2017-06-02  1.344002 -0.245280 -1.422829  0.108585\n",
       "2017-06-03  0.536841  0.015907 -0.734943  0.749065\n",
       "2017-06-04  0.988480  0.489212 -1.426875  0.164377\n",
       "2017-06-05 -0.468510 -0.979854  0.293018  0.944181\n",
       "2017-06-06  0.043697 -0.211873 -1.554627 -0.015371\n",
       "2017-06-07  0.585974  1.262029  0.403454 -0.657079\n",
       "2017-06-08  0.309760 -0.230141  1.626267  1.151040\n",
       "2017-06-09 -0.869131  0.738002  1.211384  2.257951\n",
       "2017-06-10  1.034295 -1.547727  1.515287  0.304565"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ind1 = pd.date_range('06/1/2017', periods=10)\n",
    "df1.set_index(ind1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Using Numpy functions on DataFrame\n",
    "\n",
    "Element-wise numpy functions like log, exp, sqrt, ... and various other numpy functions can be used on DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.075197</td>\n",
       "      <td>1.071788</td>\n",
       "      <td>0.346722</td>\n",
       "      <td>0.200615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.344002</td>\n",
       "      <td>0.245280</td>\n",
       "      <td>1.422829</td>\n",
       "      <td>0.108585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.536841</td>\n",
       "      <td>0.015907</td>\n",
       "      <td>0.734943</td>\n",
       "      <td>0.749065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.988480</td>\n",
       "      <td>0.489212</td>\n",
       "      <td>1.426875</td>\n",
       "      <td>0.164377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.468510</td>\n",
       "      <td>0.979854</td>\n",
       "      <td>0.293018</td>\n",
       "      <td>0.944181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.043697</td>\n",
       "      <td>0.211873</td>\n",
       "      <td>1.554627</td>\n",
       "      <td>0.015371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.585974</td>\n",
       "      <td>1.262029</td>\n",
       "      <td>0.403454</td>\n",
       "      <td>0.657079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.309760</td>\n",
       "      <td>0.230141</td>\n",
       "      <td>1.626267</td>\n",
       "      <td>1.151040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.869131</td>\n",
       "      <td>0.738002</td>\n",
       "      <td>1.211384</td>\n",
       "      <td>2.257951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.034295</td>\n",
       "      <td>1.547727</td>\n",
       "      <td>1.515287</td>\n",
       "      <td>0.304565</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C         D\n",
       "0  0.075197  1.071788  0.346722  0.200615\n",
       "1  1.344002  0.245280  1.422829  0.108585\n",
       "2  0.536841  0.015907  0.734943  0.749065\n",
       "3  0.988480  0.489212  1.426875  0.164377\n",
       "4  0.468510  0.979854  0.293018  0.944181\n",
       "5  0.043697  0.211873  1.554627  0.015371\n",
       "6  0.585974  1.262029  0.403454  0.657079\n",
       "7  0.309760  0.230141  1.626267  1.151040\n",
       "8  0.869131  0.738002  1.211384  2.257951\n",
       "9  1.034295  1.547727  1.515287  0.304565"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.abs(df1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.07519733,  1.07178771, -0.34672175,  0.2006154 ],\n",
       "       [ 1.34400207, -0.24527969, -1.42282892,  0.10858524],\n",
       "       [ 0.53684115,  0.01590705, -0.73494288,  0.74906479],\n",
       "       [ 0.98848038,  0.48921191, -1.42687493,  0.16437737],\n",
       "       [-0.46850962, -0.97985388,  0.29301751,  0.94418121],\n",
       "       [ 0.04369676, -0.21187287, -1.55462703, -0.01537086],\n",
       "       [ 0.58597426,  1.26202949,  0.40345448, -0.65707878],\n",
       "       [ 0.30976042, -0.23014127,  1.62626658,  1.15104   ],\n",
       "       [-0.86913143,  0.73800196,  1.21138375,  2.25795057],\n",
       "       [ 1.03429485, -1.54772712,  1.51528723,  0.30456533]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert to numpy array\n",
    "np.asarray(df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, we have to apply a custom logic when transforming our DataFrame. We can use `apply` function for this. It allows us to apply a function on 1D arrays to each column or row. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-0.869131</td>\n",
       "      <td>-1.547727</td>\n",
       "      <td>-1.554627</td>\n",
       "      <td>-0.657079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.344002</td>\n",
       "      <td>1.262029</td>\n",
       "      <td>1.626267</td>\n",
       "      <td>2.257951</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            A         B         C         D\n",
       "min -0.869131 -1.547727 -1.554627 -0.657079\n",
       "max  1.344002  1.262029  1.626267  2.257951"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def fn(x):\n",
    "    \"\"\"\n",
    "    Get max and min of the columns\n",
    "    \"\"\"\n",
    "    return pd.Series([x.min(), x.max()], index=['min', 'max'])\n",
    "\n",
    "df1.apply(fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Element-wise mapping can be done too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.075</td>\n",
       "      <td>1.072</td>\n",
       "      <td>-0.347</td>\n",
       "      <td>0.201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.344</td>\n",
       "      <td>-0.245</td>\n",
       "      <td>-1.423</td>\n",
       "      <td>0.109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.537</td>\n",
       "      <td>0.016</td>\n",
       "      <td>-0.735</td>\n",
       "      <td>0.749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.988</td>\n",
       "      <td>0.489</td>\n",
       "      <td>-1.427</td>\n",
       "      <td>0.164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.469</td>\n",
       "      <td>-0.980</td>\n",
       "      <td>0.293</td>\n",
       "      <td>0.944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.044</td>\n",
       "      <td>-0.212</td>\n",
       "      <td>-1.555</td>\n",
       "      <td>-0.015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.586</td>\n",
       "      <td>1.262</td>\n",
       "      <td>0.403</td>\n",
       "      <td>-0.657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.310</td>\n",
       "      <td>-0.230</td>\n",
       "      <td>1.626</td>\n",
       "      <td>1.151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-0.869</td>\n",
       "      <td>0.738</td>\n",
       "      <td>1.211</td>\n",
       "      <td>2.258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.034</td>\n",
       "      <td>-1.548</td>\n",
       "      <td>1.515</td>\n",
       "      <td>0.305</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        A       B       C       D\n",
       "0  -0.075   1.072  -0.347   0.201\n",
       "1   1.344  -0.245  -1.423   0.109\n",
       "2   0.537   0.016  -0.735   0.749\n",
       "3   0.988   0.489  -1.427   0.164\n",
       "4  -0.469  -0.980   0.293   0.944\n",
       "5   0.044  -0.212  -1.555  -0.015\n",
       "6   0.586   1.262   0.403  -0.657\n",
       "7   0.310  -0.230   1.626   1.151\n",
       "8  -0.869   0.738   1.211   2.258\n",
       "9   1.034  -1.548   1.515   0.305"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fmt = lambda x: \"{:.3f}\".format(x)\n",
    "df1.applymap(fmt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Exploratory Data Analysis using Pandas\n",
    "\n",
    "In this section, we'll demonstrate how Pandas is used for analyzing real-world datasets.\n",
    "\n",
    "**Loading data**\n",
    "\n",
    "The first thing that we have to do is acquire the data and load it into our system. For the following examples, we will use the very popular Titanic dataset. We can load the data by using one of the following commands in Pandas.\n",
    "\n",
    "|Function| Description|\n",
    "|:---|:---|\n",
    "|read_csv | Load delimited data from a file, URL, or file-like object. Use comma as default delimiter|\n",
    "|read_table | Load delimited data from a file, URL, or file-like object. Use tab ('\\\\t') as default delimiter|\n",
    "|read_fwf | Read data in fixed-width column format (that is, no delimiters)|\n",
    "|read_clipboard | Version of read_table that reads data from the clipboard. Useful for converting tables from web pages.|\n",
    "\n",
    "`train.csv` in our 'data' folder contains the \"train set\" of the titanic dataset. We'll explore what \"train set\" means in future sessions (or you could read up on your own), but what's important to know is this is simply a subset of the original dataset. We'll be loading this file for the following examples. We will be using `read_csv` here since our dataset is in csv format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/train.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we have appended `train.csv` to data. What happens here is in your current directory, you go inside 'data' folder and then access the `train.csv` file. \n",
    "\n",
    "We used the `head` method to just display the first 5 (default) entries.\n",
    "\n",
    "> To know more about what the column names mean, see this link: https://www.kaggle.com/c/titanic/data\n",
    "\n",
    "Let's check the dimensions and data types in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(891, 12)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      int64\n",
       "Survived         int64\n",
       "Pclass           int64\n",
       "Name            object\n",
       "Sex             object\n",
       "Age            float64\n",
       "SibSp            int64\n",
       "Parch            int64\n",
       "Ticket          object\n",
       "Fare           float64\n",
       "Cabin           object\n",
       "Embarked        object\n",
       "dtype: object"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We can use the following snippet to check the number of null values in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      0\n",
       "Survived         0\n",
       "Pclass           0\n",
       "Name             0\n",
       "Sex              0\n",
       "Age            177\n",
       "SibSp            0\n",
       "Parch            0\n",
       "Ticket           0\n",
       "Fare             0\n",
       "Cabin          687\n",
       "Embarked         2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, there are null values in `Age`, `Cabin` and `Embarked` columns.\n",
    "\n",
    "What if we want to get the age?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    22.0\n",
       "1    38.0\n",
       "2    26.0\n",
       "3    35.0\n",
       "4    35.0\n",
       "Name: Age, dtype: float64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Age'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the average age. Note that we can use this method for numeric data types. The different types of data (in general, not Python) will be further explained in future modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.69911764705882"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Age'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about categorical data types?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "male      577\n",
       "female    314\n",
       "Name: Sex, dtype: int64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Sex'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also filter the dataset to only those who survived."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)</td>\n",
       "      <td>female</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>347742</td>\n",
       "      <td>11.1333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Nasser, Mrs. Nicholas (Adele Achem)</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>237736</td>\n",
       "      <td>30.0708</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "8            9         1       3   \n",
       "9           10         1       2   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "8  Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)  female  27.0      0   \n",
       "9                Nasser, Mrs. Nicholas (Adele Achem)  female  14.0      1   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "8      2            347742  11.1333   NaN        S  \n",
       "9      0            237736  30.0708   NaN        C  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_survived = df[df.Survived==1]\n",
    "df_survived.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.343689655172415"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_survived['Age'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "female    233\n",
       "male      109\n",
       "Name: Sex, dtype: int64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_survived['Sex'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}