getting_started_notebooks/intro_tutorials/02_Introduction_to_cuDF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"introduction\"></a>\n",
    "## Introduction to cuDF\n",
    "#### By Paul Hendricks\n",
    "-------\n",
    "\n",
    "In this notebook, we will show how to work with cuDF DataFrames in RAPIDS.\n",
    "\n",
    "**Table of Contents**\n",
    "\n",
    "* [Introduction to cuDF](#introduction)\n",
    "* [Setup](#setup)\n",
    "* [cuDF Series Basics](#series)\n",
    "* [cuDF DataFrame Basics](#dataframes)\n",
    "* [Input/Output](#io)\n",
    "* [cuDF API](#cudfapi)\n",
    "* [Conclusion](#conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## Setup\n",
    "\n",
    "This notebook was tested using the following Docker containers:\n",
    "\n",
    "* `rapidsai/rapidsai-dev-nightly:0.10-cuda10.0-devel-ubuntu18.04-py3.7` container from [DockerHub](https://hub.docker.com/r/rapidsai/rapidsai-nightly)\n",
    "\n",
    "This notebook was run on the NVIDIA GV100 GPU. Please be aware that your system may be different and you may need to modify the code or install packages to run the below examples. \n",
    "\n",
    "If you think you have found a bug or an error, please file an issue here: https://github.com/rapidsai/notebooks-contrib/issues\n",
    "\n",
    "Before we begin, let's check out our hardware setup by running the `nvidia-smi` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's see what CUDA version we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvcc --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"series\"></a>\n",
    "## cuDF Series Basics\n",
    "\n",
    "First, let's load the cuDF library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cudf; print('cuDF Version:', cudf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two main data structures in cuDF: a `Series` object and a `DataFrame` object. Multiple `Series` objects are used as columns for a `DataFrame`. We'll first explore the `Series` class and build upon that foundation to later introduce how to work with objects of type `DataFrame`.\n",
    "\n",
    "We can create a `Series` object using the `cudf.Series` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column = cudf.Series([10, 11, 12, 13])\n",
    "column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see from the output that `column` is an object of type `cudf.Series` and has 4 rows.\n",
    "\n",
    "Another way to inspect a `Series` is to use the Python `print` statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(column)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that our `Series` object has four rows with values 10, 11, 12, and 13. We also see that the type of this data is `int64`. There are several ways to represent data using cuDF. The most common formats are `int8`, `int32`, `int64`, `float32`, and `float64`.\n",
    "\n",
    "We also see a column of values on the left hand side with values 0, 1, 2, 3. These values represent the index of the `Series`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(column.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a new column with a different index by using the `set_index` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_column = column.set_index([5, 6, 7, 8]) \n",
    "print(new_column)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indexes are useful for operations like joins and groupbys."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"dataframes\"></a>\n",
    "## cuDF DataFrame Basics\n",
    "\n",
    "As we showed in the previous tutorial, cuDF DataFrames are a tabular structure of data that reside on the GPU. We interface with these cuDF DataFrames in the same way we interface with Pandas DataFrames that reside on the CPU - with a few deviations.\n",
    "\n",
    "In the next several sections, we'll show how to create and manipulate cuDF DataFrames. For more information on using cuDF DataFrames, check out the documentation: https://rapidsai.github.io/projects/cudf/en/latest/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating a cudf DataFrame using lists\n",
    "\n",
    "There are several ways to create a cuDF DataFrame. The easiest of these is to instantiate an empty cuDF DataFrame and then use Python list objects or NumPy arrays to create columns. Below, we create an empty cuDF DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame()\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can create two columns named `key` and `value` by using the bracket notation with the cuDF DataFrame and storing either a list of Python values or a NumPy array into that column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np; print('NumPy Version:', np.__version__)\n",
    "\n",
    "\n",
    "# here we create two columns named \"key\" and \"value\"\n",
    "df['key'] = [0, 1, 2, 3, 4]\n",
    "df['value'] = np.arange(10, 15)\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating a cudf DataFrame using a list of tuples or a dictionary\n",
    "\n",
    "Another way we can create a cuDF DataFrame is by providing a mapping of column names to column values, either via a list of tuples or by using a dictionary. In the below examples, we create a list of two-value tuples; the first value is the name of the column - for example, `id` or `timestamp` - and the second value is a list of Python objects or Numpy arrays. Note that we don't have to constrain the data stored in our cuDF DataFrames to common data types like integers or floats - we can use more exotic data types such as datetimes or strings. We'll investigate how such data types behave on the GPU a bit later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime, timedelta\n",
    "\n",
    "\n",
    "ids = np.arange(5)\n",
    "t0 = datetime.strptime('2018-10-07 12:00:00', '%Y-%m-%d %H:%M:%S')\n",
    "timestamps = [(t0+ timedelta(seconds=x)) for x in range(5)]\n",
    "timestamps_np = np.array(timestamps, dtype='datetime64')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame([('id', ids), ('timestamp', timestamps_np)])\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we can create a dictonary of key-value pairs, where each key in the dictionary represents a column name and each value associated with the key represents the values that belong in that column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'id': ids, 'timestamp': timestamps_np})\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating a cudf DataFrame from a Pandas DataFrame\n",
    "\n",
    "Pandas DataFrames are a first class citizen within cuDF - this means that we can create a cuDF DataFrame from a Pandas DataFrame and vice versa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd; print('Pandas Version:', pd.__version__)\n",
    "\n",
    "\n",
    "pandas_df = pd.DataFrame({'a': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],\n",
    "                          'b': [0.0, 0.1, 0.2, None, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]})\n",
    "print(pandas_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `cudf.from_pandas` or `cudf.DataFrame.from_pandas` functions to create a cuDF DataFrame from a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.from_pandas(pandas_df)\n",
    "# df = cudf.DataFrame.from_pandas(pandas_df)  # alternative\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating a cuDF DataFrame from cuDF Series\n",
    "\n",
    "We can create a cuDF DataFrame from one or more cuDF Series objects by passing the Series objects in a dictionary mapping each Series object to a column name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column1 = cudf.Series([1, 2, 3, 4])\n",
    "column2 = cudf.Series([5, 6, 7, 8])\n",
    "column3 = cudf.Series([9, 10, 11, 12])\n",
    "df = cudf.DataFrame({'a': column1, 'b': column2, 'c': column3})\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inspecting a cuDF DataFrame\n",
    "\n",
    "There are several ways to inspect a cuDF DataFrame. The first method is to enter the cuDF DataFrame directly into the REPL. This shows us information about the type of the object, and metadata such as the number of rows or columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.arange(0, 100), 'b': np.arange(100, 0, -1)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A second way to inspect a cuDF DataFrame is to wrap the object in a Python `print` function. This results in showing the rows and columns of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For very large dataframes, we often want to see the first couple rows. We can use the `head` method of a cuDF DataFrame to view the first N rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Columns\n",
    "\n",
    "cuDF DataFrames store metadata such as information about columns or data types. We can access the columns of a cuDF DataFrame using the `.columns` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can modify the columns of a cuDF DataFrame by modifying the `columns` attribute. We can do this by setting that attribute equal to a list of strings representing the new columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns = ['c', 'd']\n",
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Types\n",
    "\n",
    "We can also inspect the data types of the columns of a cuDF DataFrame using the `dtypes` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can modify the data types of the columns of a cuDF DataFrame by passing in a cuDF Series with a modified data type. Be warned that silent errors may be introduced from nonsensical type conversations - for example, changing a float to an integer or vice versa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['c'] = df['c'].astype(np.float32)\n",
    "df['d'] = df['d'].astype(np.int32)\n",
    "print(df.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Series\n",
    "\n",
    "cuDF DataFrames are composed of rows and columns. Each column is represented using an object of type `Series`. For example, if we subset a cuDF DataFrame using just one column we will be returned an object of type `cudf.dataframe.series.Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(df['c']))\n",
    "print(df['c'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Index\n",
    "\n",
    "Like `Series` objects, each `DataFrame` has an index attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the index values to subset the `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df[df.index == 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Converting a cudf DataFrame to a Pandas DataFrame\n",
    "\n",
    "We can convert a cuDF DataFrame back to a Pandas DataFrame using the `to_pandas` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pandas_df = df.to_pandas()\n",
    "print(type(pandas_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Converting a cudf DataFrame to a NumPy Array\n",
    "\n",
    "Often we want to work with NumPy arrays. We can convert a cuDF DataFrame to a NumPy array by first converting it to a Pandas DataFrame using the `to_pandas` method followed by accessing the `values` attribute of the Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numpy_array = df.to_pandas().values\n",
    "print(type(numpy_array))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Converting a cudf DataFrame to Other Data Formats\n",
    "\n",
    "We can also convert a cuDF DataFrame to other data formats. \n",
    "\n",
    "For more information, see the documentation: https://docs.rapids.ai/api/cudf/stable/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"io\"></a>\n",
    "## Input/Output\n",
    "\n",
    "Before we process data and use it in machine learning models, we need to be able to load it into memory and write it after we're done using it. There are several ways to do this using cuDF."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Writing and Loading CSV Files\n",
    "\n",
    "At this time, there is no direct way to use to cuDF to write directly to CSV. However, we can conver the cuDF DataFrame to a Pandas DataFrame and then write it directly to a CSV."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_pandas().to_csv('./dataset.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps one of the most common ways to create cuDF DataFrames is by loading a table that is stored as a file on disk. cuDF provides a lot of functionality for reading in a variety of different data formats. Below, we show how easy it is to read in a CSV file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.read_csv('./dataset.csv')\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CSV files come in many flavors and cuDF tries to be as flexible as possible, mirroring the Pandas API wherever possible. For more information on possible parameters for working with files, see the cuDF IO documentation: \n",
    "\n",
    "https://rapidsai.github.io/projects/cudf/en/latest/api.html#cudf.io.csv.read_csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cudfapi\"></a>\n",
    "## cuDF API\n",
    "\n",
    "The cuDF API is pleasantly simple and mirrors the Pandas API as closely as possible. In this section, we will explore the cuDF API and show how to perform common data manipulation operations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selecting Rows or Columns\n",
    "\n",
    "We can select rows from a cuDF DataFrame using slicing syntax. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.arange(0, 100).astype(np.float32), \n",
    "                     'b': np.arange(100, 0, -1).astype(np.float32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several ways to select a column from a cuDF DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df['a'])\n",
    "# print(df.a)  # alternative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also select multiple columns by passing in a list of column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df[['a', 'b']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can select specific rows and columns using the slicing syntax as well as passing in a list of column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.loc[0:5, ['a']])\n",
    "# print(df.loc[0:5, ['a', 'b']])  # to select multiple columns, pass in multiple column names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Defining New Columns\n",
    "\n",
    "We often want to define new columns from existing columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.arange(0, 100).astype(np.float32), \n",
    "                     'b': np.arange(100, 0, -1).astype(np.float32), \n",
    "                     'c': np.arange(100, 200).astype(np.float32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['d'] = np.arange(200, 300).astype(np.float32)\n",
    "\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.arange(300, 400).astype(np.float32)\n",
    "df.add_column('e', data)\n",
    "\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dropping Columns\n",
    "\n",
    "Alternatively, we may want to remove columns from our `DataFrame`. We can do so using the `drop_column` method. Note that this method removes a column in-place - meaning that the `DataFrame` we act on will be modified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.arange(0, 100).astype(np.float32), \n",
    "                     'b': np.arange(100, 0, -1).astype(np.float32), \n",
    "                     'c': np.arange(100, 200).astype(np.float32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop_column('a')\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to remove a column without modifying the original DataFrame, we can use the `drop` method. This method will return a new DataFrame without that column (or columns)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.arange(0, 100).astype(np.float32), \n",
    "                     'b': np.arange(100, 0, -1).astype(np.float32), \n",
    "                     'c': np.arange(100, 200).astype(np.float32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_df = df.drop('a')\n",
    "\n",
    "print('Original DataFrame:')\n",
    "print(df)\n",
    "print(79 * '-')\n",
    "print('New DataFrame:')\n",
    "print(new_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also pass in a list of column names to drop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_df = df.drop(['a', 'b'])\n",
    "\n",
    "print('Original DataFrame:')\n",
    "print(df)\n",
    "print(79 * '-')\n",
    "print('New DataFrame:')\n",
    "print(new_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Missing Data\n",
    "\n",
    "Sometimes data is not as clean as we would like it - often there wrong values or values that are missing entirely. cuDF DataFrames can represent missing values using the Python `None` keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': [0, None, 2, 3, 4, 5, 6, 7, 8, None, 10],\n",
    "                     'b': [0.0, 0.1, 0.2, None, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], \n",
    "                     'c': [0.0, 0.1, None, None, 0.4, 0.5, None, 0.7, 0.8, 0.9, 1.0]})\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also fill in these missing values with another value using the `fillna` method. Both `Series` and `DataFrame` objects implement this method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['c'] = df['c'].fillna(999)\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_df = df.fillna(-1)\n",
    "print(new_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Boolean Indexing\n",
    "\n",
    "We previously saw how we can select certain rows from our dataset by using the bracket `[]` notation. However, we may want to select rows based on a certain criteria - this is called boolean indexing. We can combine the indexing notation with an array of boolean values to select only certain rows that meet this criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                     'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                     'c': np.arange(0, 100).astype(np.int32), \n",
    "                     'd': np.arange(100, 0, -1).astype(np.int32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = df['a'] == 3\n",
    "df[mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sorting Data\n",
    "\n",
    "Data is often not sorted before we start to work with it. Sorting data is is very useful for optimizing operations like joins and aggregations, especially when the data is distributed.\n",
    "\n",
    "We can sort data in cuDF using the `sort_values` method and passing in which column we want to sort by. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                     'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                     'c': np.arange(0, 100).astype(np.int32), \n",
    "                     'd': np.arange(100, 0, -1).astype(np.int32)})\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.sort_values('d').head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also specify if the column we're sorting should be sorted in ascending or descending order by using the `ascending` argument and passing in `True` or `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.sort_values('c', ascending=False).head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can sort by multiple columns by passing in a list of column names. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.sort_values(['a', 'b']).head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also specify which of those columns should be sorted in ascending or descending order by passing in a list of boolean values, where each boolean value maps to each column, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Sort with all columns specified descending:')\n",
    "print(df.sort_values(['a', 'b'], ascending=False).head())\n",
    "print(79 * '-')\n",
    "print('Sort with both a descending and b ascending:')\n",
    "print(df.sort_values(['a', 'b'], ascending=[False, True]).head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Statistical Operations\n",
    "\n",
    "There are several statistical operations we can use to aggregate our data in meaningful ways. These can be applied to both `Series` and `DataFrame` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                     'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                     'c': np.arange(0, 100).astype(np.int32), \n",
    "                     'd': np.arange(100, 0, -1).astype(np.int32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['a'].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Applymap Operations\n",
    "\n",
    "While cuDF allows us to define new columns in interesting ways, we often want to work with more complex functions. We can define a function and use the `applymap` method to apply this function to each value in a column in element-wise fashion. While the below example is simple, it can be very easily extended to more complex workflows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                     'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                     'c': np.arange(0, 100).astype(np.int32), \n",
    "                     'd': np.arange(100, 0, -1).astype(np.int32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_ten_to_x(x):\n",
    "    return x + 10\n",
    "\n",
    "print(df['c'].applymap(add_ten_to_x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Histogramming\n",
    "\n",
    "We can access the value counts of a column using the `value_counts` method. Note that this is typically used with columns representing discrete data i.e. integers, strings, categoricals, etc. We may not be as interested in the value counts of numerical data e.g. how often the value 2.1 appears. The results of the `value_counts` method can be used with Python plotting libraries like Matplotlib or Seaborn to generate visualizations such as histograms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                     'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                     'c': np.arange(0, 100).astype(np.int32), \n",
    "                     'd': np.arange(100, 0, -1).astype(np.int32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = df['a'].value_counts()\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Concatenations\n",
    "\n",
    "In everyday data science, we typically work with multiple sources of data and wish to combine these data into a single more meaningful representation. These operations are often called concatenations and joins. We can concatenate two or more dataframes together row-wise or column-wise by passing in a list of the dataframes to be concatenated into the `cudf.concat` function and specifying the axis along which to concatenate these dataframes.\n",
    "\n",
    "If we want to concatenate the dataframes row-wise, we can specify `axis=0`. To concatenate column-wise, we can specify `axis=1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                      'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                      'c': np.arange(0, 100).astype(np.int32), \n",
    "                      'd': np.arange(100, 0, -1).astype(np.int32)})\n",
    "df2 = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                      'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                      'c': np.arange(0, 100).astype(np.int32), \n",
    "                      'd': np.arange(100, 0, -1).astype(np.int32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.concat([df1, df2], axis=0)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                      'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                      'c': np.arange(0, 100).astype(np.int32), \n",
    "                      'd': np.arange(100, 0, -1).astype(np.int32)})\n",
    "df2 = cudf.DataFrame({'e': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                      'f': np.random.randint(2, size=100).astype(np.int32), \n",
    "                      'g': np.arange(0, 100).astype(np.int32), \n",
    "                      'h': np.arange(100, 0, -1).astype(np.int32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.concat([df1, df2], axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Joins / Merges\n",
    "\n",
    "Multiple dataframes can be joined together using a single (or multiple) column(s). There are two syntaxes for performing joins:\n",
    "\n",
    "* One can use the `DataFrame.merge` method and pass in another dataframe to join, or\n",
    "* One can use the `cudf.merge` function and pass in which dataframes to join.\n",
    "\n",
    "Both syntaxes can also be passed a list of column names to an additional keyword argument `on` - this will specify which columns the dataframes should be joined on. If this keyword is not specified, cuDF will by default join using column names that appear in both dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                      'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                      'c': np.arange(0, 100).astype(np.int32), \n",
    "                      'd': np.arange(100, 0, -1).astype(np.int32)})\n",
    "df2 = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                      'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                      'e': np.arange(0, 100).astype(np.int32), \n",
    "                      'f': np.arange(100, 0, -1).astype(np.int32)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df1.merge(df2)\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df1.merge(df2, on=['a'])\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df1.merge(df2, on=['a', 'b'])\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.merge(df1, df2)\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.merge(df1, df2, on=['a'])\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.merge(df1, df2, on=['a', 'b'])\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Groupbys\n",
    "\n",
    "A useful operation when working with datasets is to group the data using a specific key and aggregate the values mapping to those keys. For example, we might want to aggregate multiple temperature measurements taken during a day from a specific sensor and average those measurements to find avergage daily temperature at a specific geolocation.\n",
    "\n",
    "cuDF allows us to perform such an operation using the `groupby` method. This will create an object of type `cudf.groupby.groupby.Groupby` that we can operate on using aggregation functions such as `sum`, `var`, or complex aggregation functions defined by the user.\n",
    "\n",
    "We can also specify multiple columns to group on by passing a list of column names to the `groupby` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame({'a': np.repeat([0, 1, 2, 3], 25).astype(np.int32), \n",
    "                     'b': np.random.randint(2, size=100).astype(np.int32), \n",
    "                     'c': np.arange(0, 100).astype(np.int32), \n",
    "                     'd': np.arange(100, 0, -1).astype(np.int32)})\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped_df = df.groupby('a')\n",
    "grouped_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregation = grouped_df.sum()\n",
    "print(aggregation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregation = df.groupby(['a', 'b']).sum().to_pandas()\n",
    "print(aggregation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### One Hot Encoding\n",
    "\n",
    "Data scientists often work with discrete data such as integers or categories. However, this data can be represented using a One Hote Encoding format.\n",
    "\n",
    "cuDF allows us to convert these discrete datas to a One Hot Encoding format using the `one_hot_encoding` method. We can pass this method the column name to convert, a prefix with which to prepend to each newly created column, and the categories of data to create new columns for. We can pass in all the categories in the discrete data or a subset - cuDF will flexibly handle both and only create new columns for the categories specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = [0, 1, 2, 3]\n",
    "df = cudf.DataFrame({'a': np.repeat(categories, 25).astype(np.int32), \n",
    "                     'b': np.arange(0, 100).astype(np.int32), \n",
    "                     'c': np.arange(100, 0, -1).astype(np.int32)})\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = df.one_hot_encoding('a', prefix='a_', cats=categories)\n",
    "print(result.head())\n",
    "print(result.tail())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = df.one_hot_encoding('a', prefix='a_', cats=[0, 1, 2])\n",
    "print(result.head())\n",
    "print(result.tail())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"conclusion\"></a>\n",
    "## Conclusion\n",
    "\n",
    "In this notebook, we showed how to work with cuDF DataFrames in RAPIDS.\n",
    "\n",
    "To learn more about RAPIDS, be sure to check out: \n",
    "\n",
    "* [Open Source Website](http://rapids.ai)\n",
    "* [GitHub](https://github.com/rapidsai/)\n",
    "* [Press Release](https://nvidianews.nvidia.com/news/nvidia-introduces-rapids-open-source-gpu-acceleration-platform-for-large-scale-data-analytics-and-machine-learning)\n",
    "* [NVIDIA Blog](https://blogs.nvidia.com/blog/2018/10/10/rapids-data-science-open-source-community/)\n",
    "* [Developer Blog](https://devblogs.nvidia.com/gpu-accelerated-analytics-rapids/)\n",
    "* [NVIDIA Data Science Webpage](https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}