7 - GPU DataFrames.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GTC 2018 Numba Tutorial Notebook 7: GPU DataFrames\n",
    "\n",
    "## What is a DataFrame?\n",
    "\n",
    "So far, we've been using Numba to operate on objects that behave like NumPy arrays: n-dimensional where every element has the same type.  (Note: The element type can be a struct.)  For many applications, like waveforms, images, lists of numbers, etc, it is very natural to create n-dimensional arrays.\n",
    "\n",
    "However, another data model is also very common in data science.  The DataFrame is a 2-dimensional table, where each row represents a record and each column can have a different data type:\n",
    "\n",
    "| time | sensor_id | temp | humidity | flags |\n",
    "|------|-----------|------|----------|-------|\n",
    "| 1    | A         | 28   | 50       | 0     |\n",
    "| 4    | B         | 25   | 51       | 0     |\n",
    "| 8    | A         | 27   | 49       | 1     |\n",
    "\n",
    "The DataFrame is very similar to the data returned by relational databases.  DataFrames are stored such that all the data in each column is contiguous in memory, which allows for much faster columnar processing.\n",
    "\n",
    "In Python, the standard library for working with DataFrames on the CPU is Pandas.  Last year, as part of the [GPU Open Analytics Initiative](http://gpuopenanalytics.com/), the Numba developers began implementing a GPU DataFrame library, called `PyGDF`, inspired by the Pandas API and based on the [Apache Arrow](https://arrow.apache.org/) project.\n",
    "\n",
    "In this notebook, we'll introduce the capabilities of PyGDF, and show how your knowledge of CUDA Python will allow you to implement some powerful data transformations that run directly on the GPU.\n",
    "\n",
    "**WARNING**: PyGDF is still very new and experimental.  It is continually being improved and optimized, so if you encounter issues, let us know at: https://github.com/gpuopenanalytics/pygdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data\n",
    "\n",
    "Currently, loading and parsing data from disk still has to take place on the CPU.  Let's load a CSV file of sensor data with Pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time         float64\n",
       "sensor_id     object\n",
       "temp         float64\n",
       "humidity     float64\n",
       "flags          int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('sensor.csv', dtype=dict(flags=int))\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>sensor_id</th>\n",
       "      <th>temp</th>\n",
       "      <th>humidity</th>\n",
       "      <th>flags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000</td>\n",
       "      <td>B</td>\n",
       "      <td>22.498096</td>\n",
       "      <td>0.465723</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.001</td>\n",
       "      <td>C</td>\n",
       "      <td>21.693834</td>\n",
       "      <td>0.475885</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.002</td>\n",
       "      <td>B</td>\n",
       "      <td>21.130495</td>\n",
       "      <td>0.483004</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003</td>\n",
       "      <td>B</td>\n",
       "      <td>20.719901</td>\n",
       "      <td>0.488192</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.004</td>\n",
       "      <td>C</td>\n",
       "      <td>20.484617</td>\n",
       "      <td>0.491165</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    time sensor_id       temp  humidity  flags\n",
       "0  0.000         B  22.498096  0.465723      0\n",
       "1  0.001         C  21.693834  0.475885      0\n",
       "2  0.002         B  21.130495  0.483004      0\n",
       "3  0.003         B  20.719901  0.488192      0\n",
       "4  0.004         C  20.484617  0.491165      0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The format of this (not real) data is:\n",
    "\n",
    "  * time - Time since start of recording (s)\n",
    "  * sensor_id - Name of temperature/humidity sensor\n",
    "  * temp - Temperature (C)\n",
    "  * humidity - Relative humidity (0-1)\n",
    "  * flags - flag is set to 1 if motion sensor records activity, 0 otherwise\n",
    "  \n",
    "To load this data onto the GPU, we need to recode the `sensor_id` column, which is currently a string.  PyGDF does not support strings on the GPU, but fortunately we can rewrite this column as a categorical.  Categorical columns map each unique value to an integer for faster processing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CategoricalDtype(categories=['A', 'B', 'C'], ordered=False)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['sensor_id'] = df['sensor_id'].astype('category')\n",
    "df['sensor_id'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's load this data onto the GPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time          float64\n",
       "sensor_id    category\n",
       "temp          float64\n",
       "humidity      float64\n",
       "flags           int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pygdf\n",
    "\n",
    "gdf = pygdf.DataFrame.from_pandas(df)\n",
    "gdf.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to look at some part of the DataFrame on the CPU, we can call the `.to_pandas()` method to copy the data back to a regular Pandas DataFrame.  This code eshows us the first 5 rows on the GPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>sensor_id</th>\n",
       "      <th>temp</th>\n",
       "      <th>humidity</th>\n",
       "      <th>flags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000</td>\n",
       "      <td>B</td>\n",
       "      <td>22.498096</td>\n",
       "      <td>0.465723</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.001</td>\n",
       "      <td>C</td>\n",
       "      <td>21.693834</td>\n",
       "      <td>0.475885</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.002</td>\n",
       "      <td>B</td>\n",
       "      <td>21.130495</td>\n",
       "      <td>0.483004</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003</td>\n",
       "      <td>B</td>\n",
       "      <td>20.719901</td>\n",
       "      <td>0.488192</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.004</td>\n",
       "      <td>C</td>\n",
       "      <td>20.484617</td>\n",
       "      <td>0.491165</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    time sensor_id       temp  humidity  flags\n",
       "0  0.000         B  22.498096  0.465723      0\n",
       "1  0.001         C  21.693834  0.475885      0\n",
       "2  0.002         B  21.130495  0.483004      0\n",
       "3  0.003         B  20.719901  0.488192      0\n",
       "4  0.004         C  20.484617  0.491165      0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gdf.head().to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With suitable plotting tools [Holoviews + Datashader](http://holoviews.org/user_guide/Large_Data.html) we could plot all 1 million points in the notebook, but for simplicity we've included the plot images below:\n",
    "\n",
    "<img src=\"img/sensor_temp.png\" alt=\"temperature\" style=\"width: 250px; display: inline\"/><img src=\"img/sensor_humidity.png\" alt=\"humidity\" style=\"width: 250px; display: inline\"/>\n",
    "\n",
    "We can see glitches in the temperature data (which we will assume are due to sensor errors), which we will clean up later in this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Column Transformations\n",
    "\n",
    "One of the most basic GDF operations is a column transform.  We can perform built-in arithmetic operations on each column, such as type casting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time          float64\n",
       "sensor_id    category\n",
       "temp          float32\n",
       "humidity      float32\n",
       "flags           int32\n",
       "dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gdf['temp'] = gdf['temp'].astype(np.float32)\n",
    "gdf['humidity'] = gdf['humidity'].astype(np.float32)\n",
    "gdf['flags'] = gdf['flags'].astype(np.int32)\n",
    "gdf.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or column arithmetic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>sensor_id</th>\n",
       "      <th>temp</th>\n",
       "      <th>humidity</th>\n",
       "      <th>flags</th>\n",
       "      <th>temp_f</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000</td>\n",
       "      <td>B</td>\n",
       "      <td>22.498096</td>\n",
       "      <td>0.465723</td>\n",
       "      <td>0</td>\n",
       "      <td>72.496574</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.001</td>\n",
       "      <td>C</td>\n",
       "      <td>21.693834</td>\n",
       "      <td>0.475885</td>\n",
       "      <td>0</td>\n",
       "      <td>71.048904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.002</td>\n",
       "      <td>B</td>\n",
       "      <td>21.130495</td>\n",
       "      <td>0.483004</td>\n",
       "      <td>0</td>\n",
       "      <td>70.034889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003</td>\n",
       "      <td>B</td>\n",
       "      <td>20.719902</td>\n",
       "      <td>0.488192</td>\n",
       "      <td>0</td>\n",
       "      <td>69.295822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.004</td>\n",
       "      <td>C</td>\n",
       "      <td>20.484617</td>\n",
       "      <td>0.491165</td>\n",
       "      <td>0</td>\n",
       "      <td>68.872314</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    time sensor_id       temp  humidity  flags     temp_f\n",
       "0  0.000         B  22.498096  0.465723      0  72.496574\n",
       "1  0.001         C  21.693834  0.475885      0  71.048904\n",
       "2  0.002         B  21.130495  0.483004      0  70.034889\n",
       "3  0.003         B  20.719902  0.488192      0  69.295822\n",
       "4  0.004         C  20.484617  0.491165      0  68.872314"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gdf['temp_f'] = gdf['temp'] * 9/5 + 32\n",
    "gdf.head().to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But for non-trivial expressions, we should use the `apply_rows` function to provide a custom function that loops over all rows.  This function looks like writing serial code, but it will be translated by PyGDF (using Numba) into properly strided, parallel GPU code.  It is important that this function only operate on one row at a time, or the translation will not be correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_fahrenheit(temp, temp_f_int):\n",
    "    for i, t in enumerate(temp):\n",
    "        temp_f_int[i] = 9/5 * t + 32.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.apply_rows()` function takes the serial function, the names of the input columns, a dictionary of output columns and target data types, and optionally a dictionary of extra, non-GDF arguments (not used here):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>sensor_id</th>\n",
       "      <th>temp</th>\n",
       "      <th>humidity</th>\n",
       "      <th>flags</th>\n",
       "      <th>temp_f</th>\n",
       "      <th>temp_f_int</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000</td>\n",
       "      <td>B</td>\n",
       "      <td>22.498096</td>\n",
       "      <td>0.465723</td>\n",
       "      <td>0</td>\n",
       "      <td>72.496574</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.001</td>\n",
       "      <td>C</td>\n",
       "      <td>21.693834</td>\n",
       "      <td>0.475885</td>\n",
       "      <td>0</td>\n",
       "      <td>71.048904</td>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.002</td>\n",
       "      <td>B</td>\n",
       "      <td>21.130495</td>\n",
       "      <td>0.483004</td>\n",
       "      <td>0</td>\n",
       "      <td>70.034889</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003</td>\n",
       "      <td>B</td>\n",
       "      <td>20.719902</td>\n",
       "      <td>0.488192</td>\n",
       "      <td>0</td>\n",
       "      <td>69.295822</td>\n",
       "      <td>69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.004</td>\n",
       "      <td>C</td>\n",
       "      <td>20.484617</td>\n",
       "      <td>0.491165</td>\n",
       "      <td>0</td>\n",
       "      <td>68.872314</td>\n",
       "      <td>68</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    time sensor_id       temp  humidity  flags     temp_f  temp_f_int\n",
       "0  0.000         B  22.498096  0.465723      0  72.496574          72\n",
       "1  0.001         C  21.693834  0.475885      0  71.048904          71\n",
       "2  0.002         B  21.130495  0.483004      0  70.034889          70\n",
       "3  0.003         B  20.719902  0.488192      0  69.295822          69\n",
       "4  0.004         C  20.484617  0.491165      0  68.872314          68"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gdf = gdf.apply_rows(to_fahrenheit,\n",
    "               incols=['temp'],\n",
    "               outcols=dict(temp_f_int=np.int32),\n",
    "               kwargs={})\n",
    "gdf.head().to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtering\n",
    "\n",
    "Filtering can be done with the `query()` method, which takes an expression string of column names.  Let's remove all the rows where the flags (indicating motion near the sensor) are zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original len: 1000000\n",
      "Reduced len: 999889\n"
     ]
    }
   ],
   "source": [
    "print('Original len:', len(gdf))\n",
    "\n",
    "reduced_gdf = gdf.query('flags == 0')\n",
    "# don't need these columns anymore\n",
    "del reduced_gdf['flags']  \n",
    "del reduced_gdf['temp_f']\n",
    "del reduced_gdf['temp_f_int']\n",
    "\n",
    "print('Reduced len:', len(reduced_gdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Groupby\n",
    "\n",
    "Probably one of the most effective uses of the GPU DataFrame is for grouping operations followed by transformations or reductions.\n",
    "\n",
    "### Groupby Reductions\n",
    "\n",
    "The simplest usage of groupby is to apply a reduction to combine rows in the same group.\n",
    "\n",
    "For this analysis, we are going to look at data in 5 second time bins.  First we need to compute a bin number based on the time column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "reduced_gdf['time_bin'] = (reduced_gdf['time'] / 5).astype(np.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's compute the mean in each bin for each sensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_bin</th>\n",
       "      <th>sensor_id</th>\n",
       "      <th>time</th>\n",
       "      <th>temp</th>\n",
       "      <th>humidity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>2.500937</td>\n",
       "      <td>21.752899</td>\n",
       "      <td>0.475139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>B</td>\n",
       "      <td>2.479421</td>\n",
       "      <td>21.710459</td>\n",
       "      <td>0.475675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>C</td>\n",
       "      <td>2.517933</td>\n",
       "      <td>21.684062</td>\n",
       "      <td>0.476009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "      <td>7.571181</td>\n",
       "      <td>21.675455</td>\n",
       "      <td>0.476118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>B</td>\n",
       "      <td>7.466885</td>\n",
       "      <td>21.654796</td>\n",
       "      <td>0.476379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>C</td>\n",
       "      <td>7.463490</td>\n",
       "      <td>21.624590</td>\n",
       "      <td>0.476760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>A</td>\n",
       "      <td>12.499554</td>\n",
       "      <td>21.450349</td>\n",
       "      <td>0.478962</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>B</td>\n",
       "      <td>12.507045</td>\n",
       "      <td>21.567419</td>\n",
       "      <td>0.477483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>C</td>\n",
       "      <td>12.491396</td>\n",
       "      <td>21.479624</td>\n",
       "      <td>0.478592</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>3</td>\n",
       "      <td>A</td>\n",
       "      <td>17.502512</td>\n",
       "      <td>21.307641</td>\n",
       "      <td>0.480765</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   time_bin sensor_id       time       temp  humidity\n",
       "0         0         A   2.500937  21.752899  0.475139\n",
       "1         0         B   2.479421  21.710459  0.475675\n",
       "2         0         C   2.517933  21.684062  0.476009\n",
       "3         1         A   7.571181  21.675455  0.476118\n",
       "4         1         B   7.466885  21.654796  0.476379\n",
       "5         1         C   7.463490  21.624590  0.476760\n",
       "6         2         A  12.499554  21.450349  0.478962\n",
       "7         2         B  12.507045  21.567419  0.477483\n",
       "8         2         C  12.491396  21.479624  0.478592\n",
       "9         3         A  17.502512  21.307641  0.480765"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_gdf = reduced_gdf.groupby(by=['time_bin', 'sensor_id']).mean()\n",
    "mean_gdf.head(10).to_pandas()  # the result of a groupby is another GPU DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, we want to apply a different reduction to each column and use more than one reductions of some columns.  Let's keep the earliest time in each bin, and get the mean and standard deviation of the temp and humidity columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/seibert/anaconda/envs/gtc2018/lib/python3.6/site-packages/pygdf/numerical.py:166: RuntimeWarning: divide by zero encountered in double_scalars\n",
      "  var = asum / div - (mu ** 2) * n / div\n",
      "/home/seibert/anaconda/envs/gtc2018/lib/python3.6/site-packages/pygdf/numerical.py:166: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  var = asum / div - (mu ** 2) * n / div\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_bin</th>\n",
       "      <th>sensor_id</th>\n",
       "      <th>time</th>\n",
       "      <th>temp_mean</th>\n",
       "      <th>temp_std</th>\n",
       "      <th>humidity_mean</th>\n",
       "      <th>humidity_std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.005000</td>\n",
       "      <td>21.752899</td>\n",
       "      <td>1.665295</td>\n",
       "      <td>0.475139</td>\n",
       "      <td>0.021042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>B</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>21.710459</td>\n",
       "      <td>1.657004</td>\n",
       "      <td>0.475675</td>\n",
       "      <td>0.020937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>C</td>\n",
       "      <td>0.001000</td>\n",
       "      <td>21.684062</td>\n",
       "      <td>1.612681</td>\n",
       "      <td>0.476009</td>\n",
       "      <td>0.020377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "      <td>5.007005</td>\n",
       "      <td>21.675455</td>\n",
       "      <td>1.650455</td>\n",
       "      <td>0.476118</td>\n",
       "      <td>0.020854</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>B</td>\n",
       "      <td>5.002005</td>\n",
       "      <td>21.654796</td>\n",
       "      <td>1.638052</td>\n",
       "      <td>0.476379</td>\n",
       "      <td>0.020698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>C</td>\n",
       "      <td>5.000005</td>\n",
       "      <td>21.624590</td>\n",
       "      <td>1.703364</td>\n",
       "      <td>0.476760</td>\n",
       "      <td>0.021523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>A</td>\n",
       "      <td>10.000010</td>\n",
       "      <td>21.450349</td>\n",
       "      <td>1.678541</td>\n",
       "      <td>0.478962</td>\n",
       "      <td>0.021209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>B</td>\n",
       "      <td>10.001010</td>\n",
       "      <td>21.567419</td>\n",
       "      <td>1.646382</td>\n",
       "      <td>0.477483</td>\n",
       "      <td>0.020803</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>C</td>\n",
       "      <td>10.003010</td>\n",
       "      <td>21.479624</td>\n",
       "      <td>1.638245</td>\n",
       "      <td>0.478592</td>\n",
       "      <td>0.020700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>3</td>\n",
       "      <td>A</td>\n",
       "      <td>15.000015</td>\n",
       "      <td>21.307641</td>\n",
       "      <td>1.696614</td>\n",
       "      <td>0.480765</td>\n",
       "      <td>0.021438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>3</td>\n",
       "      <td>B</td>\n",
       "      <td>15.009015</td>\n",
       "      <td>21.328939</td>\n",
       "      <td>1.695805</td>\n",
       "      <td>0.480598</td>\n",
       "      <td>0.021093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>3</td>\n",
       "      <td>C</td>\n",
       "      <td>15.001015</td>\n",
       "      <td>21.286799</td>\n",
       "      <td>1.670114</td>\n",
       "      <td>0.481029</td>\n",
       "      <td>0.021103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>4</td>\n",
       "      <td>A</td>\n",
       "      <td>20.006020</td>\n",
       "      <td>21.026391</td>\n",
       "      <td>1.666073</td>\n",
       "      <td>0.484319</td>\n",
       "      <td>0.021052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>4</td>\n",
       "      <td>B</td>\n",
       "      <td>20.001020</td>\n",
       "      <td>21.046704</td>\n",
       "      <td>1.707032</td>\n",
       "      <td>0.484062</td>\n",
       "      <td>0.021569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>4</td>\n",
       "      <td>C</td>\n",
       "      <td>20.000020</td>\n",
       "      <td>20.975050</td>\n",
       "      <td>1.709057</td>\n",
       "      <td>0.484968</td>\n",
       "      <td>0.021595</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>5</td>\n",
       "      <td>A</td>\n",
       "      <td>25.005025</td>\n",
       "      <td>20.702156</td>\n",
       "      <td>1.656632</td>\n",
       "      <td>0.488416</td>\n",
       "      <td>0.020933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>5</td>\n",
       "      <td>B</td>\n",
       "      <td>25.001025</td>\n",
       "      <td>20.673457</td>\n",
       "      <td>1.676951</td>\n",
       "      <td>0.488779</td>\n",
       "      <td>0.021189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>5</td>\n",
       "      <td>C</td>\n",
       "      <td>25.000025</td>\n",
       "      <td>20.704019</td>\n",
       "      <td>1.627759</td>\n",
       "      <td>0.488392</td>\n",
       "      <td>0.020568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>6</td>\n",
       "      <td>A</td>\n",
       "      <td>30.001030</td>\n",
       "      <td>20.234756</td>\n",
       "      <td>1.651425</td>\n",
       "      <td>0.494322</td>\n",
       "      <td>0.020867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>6</td>\n",
       "      <td>B</td>\n",
       "      <td>30.000030</td>\n",
       "      <td>20.307624</td>\n",
       "      <td>1.719318</td>\n",
       "      <td>0.493401</td>\n",
       "      <td>0.021725</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    time_bin sensor_id       time  temp_mean  temp_std  humidity_mean  \\\n",
       "0          0         A   0.005000  21.752899  1.665295       0.475139   \n",
       "1          0         B   0.000000  21.710459  1.657004       0.475675   \n",
       "2          0         C   0.001000  21.684062  1.612681       0.476009   \n",
       "3          1         A   5.007005  21.675455  1.650455       0.476118   \n",
       "4          1         B   5.002005  21.654796  1.638052       0.476379   \n",
       "5          1         C   5.000005  21.624590  1.703364       0.476760   \n",
       "6          2         A  10.000010  21.450349  1.678541       0.478962   \n",
       "7          2         B  10.001010  21.567419  1.646382       0.477483   \n",
       "8          2         C  10.003010  21.479624  1.638245       0.478592   \n",
       "9          3         A  15.000015  21.307641  1.696614       0.480765   \n",
       "10         3         B  15.009015  21.328939  1.695805       0.480598   \n",
       "11         3         C  15.001015  21.286799  1.670114       0.481029   \n",
       "12         4         A  20.006020  21.026391  1.666073       0.484319   \n",
       "13         4         B  20.001020  21.046704  1.707032       0.484062   \n",
       "14         4         C  20.000020  20.975050  1.709057       0.484968   \n",
       "15         5         A  25.005025  20.702156  1.656632       0.488416   \n",
       "16         5         B  25.001025  20.673457  1.676951       0.488779   \n",
       "17         5         C  25.000025  20.704019  1.627759       0.488392   \n",
       "18         6         A  30.001030  20.234756  1.651425       0.494322   \n",
       "19         6         B  30.000030  20.307624  1.719318       0.493401   \n",
       "\n",
       "    humidity_std  \n",
       "0       0.021042  \n",
       "1       0.020937  \n",
       "2       0.020377  \n",
       "3       0.020854  \n",
       "4       0.020698  \n",
       "5       0.021523  \n",
       "6       0.021209  \n",
       "7       0.020803  \n",
       "8       0.020700  \n",
       "9       0.021438  \n",
       "10      0.021093  \n",
       "11      0.021103  \n",
       "12      0.021052  \n",
       "13      0.021569  \n",
       "14      0.021595  \n",
       "15      0.020933  \n",
       "16      0.021189  \n",
       "17      0.020568  \n",
       "18      0.020867  \n",
       "19      0.021725  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "binned_gdf = reduced_gdf.groupby(by=['time_bin', 'sensor_id']).agg({\n",
    "        'time': 'min',\n",
    "        'temp': ['mean', 'std'], \n",
    "        'humidity': ['mean', 'std']\n",
    "    })\n",
    "\n",
    "binned_gdf.head(20).to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group Transformations\n",
    "\n",
    "Now that we've learned some CUDA programming, we can write a custom function that `.apply_grouped()` can execute on the GPU so that each CUDA block is given an entire group to operate on.  The function will be embedded inside a CUDA kernel that assigns each group in the dataset to a CUDA block, and then you can do any collective processing of the group with the standard CUDA thread techniques that were shown in the previous notebooks.\n",
    "\n",
    "For this example, we're going to identify the \"glitch\" temperature readings by computing the mean and standard deviation of samples in the time bin (for each sensor) and then tagging as bad all records where the temperature is more than 5 standard deviations from the mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_bin</th>\n",
       "      <th>sensor_id</th>\n",
       "      <th>time</th>\n",
       "      <th>temp</th>\n",
       "      <th>humidity</th>\n",
       "      <th>bad</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.005</td>\n",
       "      <td>22.335522</td>\n",
       "      <td>0.467777</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.007</td>\n",
       "      <td>20.628517</td>\n",
       "      <td>0.489346</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.012</td>\n",
       "      <td>21.175602</td>\n",
       "      <td>0.482434</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.013</td>\n",
       "      <td>22.518532</td>\n",
       "      <td>0.465465</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.017</td>\n",
       "      <td>21.654493</td>\n",
       "      <td>0.476383</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.018</td>\n",
       "      <td>19.024134</td>\n",
       "      <td>0.509619</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.021</td>\n",
       "      <td>20.450075</td>\n",
       "      <td>0.491601</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.024</td>\n",
       "      <td>20.917501</td>\n",
       "      <td>0.485695</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.025</td>\n",
       "      <td>22.370373</td>\n",
       "      <td>0.467337</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.027</td>\n",
       "      <td>22.191050</td>\n",
       "      <td>0.469603</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.028</td>\n",
       "      <td>20.376501</td>\n",
       "      <td>0.492531</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.031</td>\n",
       "      <td>22.149179</td>\n",
       "      <td>0.470132</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.034</td>\n",
       "      <td>20.488441</td>\n",
       "      <td>0.491116</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.035</td>\n",
       "      <td>20.431847</td>\n",
       "      <td>0.491831</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.037</td>\n",
       "      <td>16.041185</td>\n",
       "      <td>0.547310</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.038</td>\n",
       "      <td>21.653166</td>\n",
       "      <td>0.476399</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.042</td>\n",
       "      <td>21.375425</td>\n",
       "      <td>0.479909</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.043</td>\n",
       "      <td>21.344509</td>\n",
       "      <td>0.480299</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.045</td>\n",
       "      <td>19.372561</td>\n",
       "      <td>0.505216</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.046</td>\n",
       "      <td>21.506521</td>\n",
       "      <td>0.478252</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    time_bin sensor_id   time       temp  humidity  bad\n",
       "0          0         A  0.005  22.335522  0.467777    0\n",
       "1          0         A  0.007  20.628517  0.489346    0\n",
       "2          0         A  0.012  21.175602  0.482434    0\n",
       "3          0         A  0.013  22.518532  0.465465    0\n",
       "4          0         A  0.017  21.654493  0.476383    0\n",
       "5          0         A  0.018  19.024134  0.509619    0\n",
       "6          0         A  0.021  20.450075  0.491601    0\n",
       "7          0         A  0.024  20.917501  0.485695    0\n",
       "8          0         A  0.025  22.370373  0.467337    0\n",
       "9          0         A  0.027  22.191050  0.469603    0\n",
       "10         0         A  0.028  20.376501  0.492531    0\n",
       "11         0         A  0.031  22.149179  0.470132    0\n",
       "12         0         A  0.034  20.488441  0.491116    0\n",
       "13         0         A  0.035  20.431847  0.491831    0\n",
       "14         0         A  0.037  16.041185  0.547310    0\n",
       "15         0         A  0.038  21.653166  0.476399    0\n",
       "16         0         A  0.042  21.375425  0.479909    0\n",
       "17         0         A  0.043  21.344509  0.480299    0\n",
       "18         0         A  0.045  19.372561  0.505216    0\n",
       "19         0         A  0.046  21.506521  0.478252    0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import math\n",
    "from numba import cuda\n",
    "from numba import float32\n",
    "\n",
    "def transform_function(time_bin, sensor_id, temp, bad):\n",
    "    # each of these arguments is a NumPy view including just the elements of the current group\n",
    "    # time_bin.size == sensor_id.size == temp.size == bad.size\n",
    "    # bad will be our output array flagging\n",
    "    \n",
    "    # we need some scratch space that all threads in this block can see\n",
    "    \n",
    "    # zeroth, first, and second moments\n",
    "    moments = cuda.shared.array(3, float32)\n",
    "    # mean (element 0) and standard deviation (element 1)\n",
    "    mean_std = cuda.shared.array(2, float32)\n",
    "    \n",
    "    # Initialize the shared memory we're about to modify\n",
    "    if cuda.threadIdx.x == 0:\n",
    "        moments[0] = 0.0\n",
    "        moments[1] = 0.0\n",
    "        moments[2] = 0.0\n",
    "    cuda.syncthreads()\n",
    "    \n",
    "    # First pass: Compute temperature moments in the group\n",
    "    for i in range(cuda.threadIdx.x, temp.size, cuda.blockDim.x):\n",
    "        cuda.atomic.add(moments, 0, 1)\n",
    "        cuda.atomic.add(moments, 1, temp[i])\n",
    "        cuda.atomic.add(moments, 2, temp[i]**2)\n",
    "    cuda.syncthreads()\n",
    "\n",
    "    # Now compute mean and standard deviation\n",
    "    if cuda.threadIdx.x == 0:\n",
    "        mean_std[0] = moments[1] / moments[0]\n",
    "        mean_std[1] = math.sqrt(moments[2]/moments[0] - mean_std[0]**2)\n",
    "    cuda.syncthreads()\n",
    "\n",
    "    # Second pass: Tag bad temperatures\n",
    "    for i in range(cuda.threadIdx.x, temp.size, cuda.blockDim.x):\n",
    "        bad[i] = (abs(temp[i] - mean_std[0]) / mean_std[1]) > 5\n",
    "\n",
    "# Execute the group transformation kernel\n",
    "bad_tagged_gdf = reduced_gdf.groupby(by=['time_bin', 'sensor_id']).apply_grouped(\n",
    "    transform_function,\n",
    "    incols=['time_bin', 'sensor_id', 'temp'],\n",
    "    outcols={'bad': np.int32},\n",
    "    # Set the thread-per-block for the GPU kernel.\n",
    "    # It's defaulted to 1 to emulate serial behavior.\n",
    "    tpb=32,\n",
    ")\n",
    "bad_tagged_gdf.head(20).to_pandas() # most of these will be good values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to see just the temperature samples that failed the test, we can select them out with `query()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "604430    48.201454\n",
       "878804    48.067596\n",
       "702695    47.989872\n",
       "888253    47.676571\n",
       "733252    47.316669\n",
       "609146    47.278095\n",
       "842673    47.066006\n",
       "523866    46.372456\n",
       "608021    45.988064\n",
       "843641    45.987778\n",
       "dtype: float32"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bad_tagged_gdf.query('bad == 1').temp.nlargest(10).to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "620991    39.570740\n",
       "645698    38.800659\n",
       "744413    38.731720\n",
       "740898    38.565994\n",
       "632257    38.479790\n",
       "989006    38.455040\n",
       "621341    38.163860\n",
       "619041    38.061131\n",
       "640830    37.938618\n",
       "996812    37.926849\n",
       "dtype: float32"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bad_tagged_gdf.query('bad == 0').temp.nlargest(10).to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can compute our mean temperature using only the good measurements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_bin</th>\n",
       "      <th>sensor_id</th>\n",
       "      <th>time</th>\n",
       "      <th>temp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>2.500937</td>\n",
       "      <td>21.752899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>B</td>\n",
       "      <td>2.479421</td>\n",
       "      <td>21.710459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>C</td>\n",
       "      <td>2.517933</td>\n",
       "      <td>21.684062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "      <td>7.571181</td>\n",
       "      <td>21.675455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>B</td>\n",
       "      <td>7.466885</td>\n",
       "      <td>21.654796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>C</td>\n",
       "      <td>7.463490</td>\n",
       "      <td>21.624590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>A</td>\n",
       "      <td>12.499554</td>\n",
       "      <td>21.450349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>B</td>\n",
       "      <td>12.507045</td>\n",
       "      <td>21.567419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>C</td>\n",
       "      <td>12.491396</td>\n",
       "      <td>21.479624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>3</td>\n",
       "      <td>A</td>\n",
       "      <td>17.502512</td>\n",
       "      <td>21.307641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>3</td>\n",
       "      <td>B</td>\n",
       "      <td>17.524434</td>\n",
       "      <td>21.321636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>3</td>\n",
       "      <td>C</td>\n",
       "      <td>17.469235</td>\n",
       "      <td>21.286799</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>4</td>\n",
       "      <td>A</td>\n",
       "      <td>22.448523</td>\n",
       "      <td>21.026391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>4</td>\n",
       "      <td>B</td>\n",
       "      <td>22.486833</td>\n",
       "      <td>21.046704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>4</td>\n",
       "      <td>C</td>\n",
       "      <td>22.561581</td>\n",
       "      <td>20.975050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>5</td>\n",
       "      <td>A</td>\n",
       "      <td>27.520496</td>\n",
       "      <td>20.702156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>5</td>\n",
       "      <td>B</td>\n",
       "      <td>27.483622</td>\n",
       "      <td>20.673457</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>5</td>\n",
       "      <td>C</td>\n",
       "      <td>27.492102</td>\n",
       "      <td>20.704019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>6</td>\n",
       "      <td>A</td>\n",
       "      <td>32.461595</td>\n",
       "      <td>20.234756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>6</td>\n",
       "      <td>B</td>\n",
       "      <td>32.478791</td>\n",
       "      <td>20.307624</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    time_bin sensor_id       time       temp\n",
       "0          0         A   2.500937  21.752899\n",
       "1          0         B   2.479421  21.710459\n",
       "2          0         C   2.517933  21.684062\n",
       "3          1         A   7.571181  21.675455\n",
       "4          1         B   7.466885  21.654796\n",
       "5          1         C   7.463490  21.624590\n",
       "6          2         A  12.499554  21.450349\n",
       "7          2         B  12.507045  21.567419\n",
       "8          2         C  12.491396  21.479624\n",
       "9          3         A  17.502512  21.307641\n",
       "10         3         B  17.524434  21.321636\n",
       "11         3         C  17.469235  21.286799\n",
       "12         4         A  22.448523  21.026391\n",
       "13         4         B  22.486833  21.046704\n",
       "14         4         C  22.561581  20.975050\n",
       "15         5         A  27.520496  20.702156\n",
       "16         5         B  27.483622  20.673457\n",
       "17         5         C  27.492102  20.704019\n",
       "18         6         A  32.461595  20.234756\n",
       "19         6         B  32.478791  20.307624"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_group = bad_tagged_gdf.query('bad == 0').groupby(['time_bin', 'sensor_id']).agg(dict(time='min', temp='mean'))\n",
    "filtered_group.head(20).to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}