02-collections.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border: 2px solid white;\">\n",
       "<tr>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Client</h3>\n",
       "<ul style=\"text-align: left; list-style: none; margin: 0; padding: 0;\">\n",
       "  <li><b>Scheduler: </b>tcp://127.0.0.1:42581</li>\n",
       "  <li><b>Dashboard: </b><a href='http://127.0.0.1:8787/status' target='_blank'>http://127.0.0.1:8787/status</a></li>\n",
       "</ul>\n",
       "</td>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Cluster</h3>\n",
       "<ul style=\"text-align: left; list-style:none; margin: 0; padding: 0;\">\n",
       "  <li><b>Workers: </b>4</li>\n",
       "  <li><b>Cores: </b>4</li>\n",
       "  <li><b>Memory: </b>2.08 GB</li>\n",
       "</ul>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Client: 'tcp://127.0.0.1:42581' processes=4 threads=4, memory=2.08 GB>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dask.distributed import Client\n",
    "\n",
    "client = Client(n_workers=4)\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bag: Parallel Lists for semi-structured data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Dask-bag excels in processing data that can be represented as a sequence of arbitrary inputs. We'll refer to this as \"messy\" data, because it can contain complex nested structures, missing fields, mixtures of data types, etc. The *functional* programming style fits very nicely with standard Python iteration, such as can be found in the `itertools` module.\n",
    "\n",
    "Dask.bag is a high level Dask collection to automate common workloads of this form.  In a nutshell\n",
    "\n",
    "    dask.bag = map, filter, toolz + parallel execution\n",
    "    \n",
    "Bags provide very general computation (any Python function.)  This generality\n",
    "comes at cost: bag operations tend to be slower than array/dataframe computations in the same way that Python tends to be slower than NumPy/Pandas\n",
    "    \n",
    "**Related Documentation**\n",
    "\n",
    "* [Bag documentation](https://docs.dask.org/en/latest/bag.html)\n",
    "* [Bag screencast](https://youtu.be/-qIiJ1XtSv0)\n",
    "* [Bag API](https://docs.dask.org/en/latest/bag-api.html)\n",
    "* [Bag examples](https://examples.dask.org/bag.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll make a random set of record data and store it to disk as many JSON files, to showcase using bag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/srv/data/0.json',\n",
       " '/srv/data/1.json',\n",
       " '/srv/data/2.json',\n",
       " '/srv/data/3.json',\n",
       " '/srv/data/4.json',\n",
       " '/srv/data/5.json',\n",
       " '/srv/data/6.json',\n",
       " '/srv/data/7.json',\n",
       " '/srv/data/8.json',\n",
       " '/srv/data/9.json']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import dask\n",
    "import json\n",
    "import os\n",
    "\n",
    "os.makedirs('data', exist_ok=True)              # Create data/ directory\n",
    "\n",
    "b = dask.datasets.make_people()                 # Make records of people\n",
    "b.map(json.dumps).to_textfiles('data/*.json')   # Encode as JSON, write to disk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bag creation\n",
    "\n",
    "We'll make a dask bag from the data files we wrote above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dask.bag<loads, npartitions=10>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import dask.bag as db\n",
    "import json\n",
    "\n",
    "b = db.read_text('data/*.json').map(json.loads)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'age': 26,\n",
       "  'name': ['Harold', 'Wilder'],\n",
       "  'occupation': 'Warehousewoman',\n",
       "  'telephone': '+1-(868)-994-8794',\n",
       "  'address': {'address': '339 Roscoe Highway', 'city': 'Temple City'},\n",
       "  'credit-card': {'number': '2700 8588 9250 6999',\n",
       "   'expiration-date': '07/22'}},)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.take(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bag transformation\n",
    "Showcasing a few map, filter, aggregate operations on our bag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'age': 40,\n",
       "  'name': ['Rachal', 'Keller'],\n",
       "  'occupation': 'Cleaner',\n",
       "  'telephone': '+1-(251)-584-6654',\n",
       "  'address': {'address': '1129 Minerva Turnpike', 'city': 'Port Hueneme'},\n",
       "  'credit-card': {'number': '2432 2984 4891 1861',\n",
       "   'expiration-date': '07/24'}},\n",
       " {'age': 58,\n",
       "  'name': ['Carey', 'Nichols'],\n",
       "  'occupation': 'Welfare Assistant',\n",
       "  'telephone': '1-604-537-4822',\n",
       "  'address': {'address': '335 May Bend', 'city': 'Longmont'},\n",
       "  'credit-card': {'number': '3771 812216 78553', 'expiration-date': '06/22'}})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.filter(lambda record: record['age'] > 30).take(2)  # Select only people over 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Warehousewoman', 'Cleaner')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.map(lambda record: record['occupation']).take(2)  # Select the occupation field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.count().compute()  # Count total number of records"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of these operations can be taken lazily, and chained together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dask.bag<topk-aggregate, npartitions=1>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = (b.filter(lambda record: record['age'] > 30)\n",
    "           .map(lambda record: record['occupation'])\n",
    "           .frequencies(sort=True)\n",
    "           .topk(10, key=1))\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# result.visualize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Press Setter', 14),\n",
       " ('Acupuncturist', 14),\n",
       " ('Ventriloquist', 14),\n",
       " ('Building Inspector', 13),\n",
       " ('Parts Supervisor', 13),\n",
       " ('Administration Assistant', 13),\n",
       " ('HGV Driver', 13),\n",
       " ('Yard Manager', 13),\n",
       " ('Palaeontologist', 13),\n",
       " ('Betting Shop', 13)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://docs.dask.org/en/latest/_images/dask-array-black-text.svg\" width=\"25%\" align=\"right\">\n",
    "Dask array provides a parallel, larger-than-memory, n-dimensional array using blocked algorithms. Simply put: distributed Numpy.\n",
    "\n",
    "*  **Parallel**: Uses all of the cores on your computer\n",
    "*  **Larger-than-memory**:  Lets you work on datasets that are larger than your available memory by breaking up your array into many small pieces, operating on those pieces in an order that minimizes the memory footprint of your computation, and effectively streaming data from disk.\n",
    "*  **Blocked Algorithms**:  Perform large computations by performing many smaller computations\n",
    "\n",
    "\n",
    "**Related Documentation**\n",
    "\n",
    "* [Array documentation](https://docs.dask.org/en/latest/array.html)\n",
    "* [Array screencast](https://youtu.be/9h_61hXCDuI)\n",
    "* [Array API](https://docs.dask.org/en/latest/array-api.html)\n",
    "* [Array examples](https://examples.dask.org/array.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.  Construct a 20000x20000 array of normally distributed random values broken up into 1000x1000 sized chunks\n",
    "2.  Take the mean along one axis\n",
    "3.  Take every 100th element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr>\n",
       "<td>\n",
       "<table>\n",
       "  <thead>\n",
       "    <tr><td> </td><th> Array </th><th> Chunk </th></tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr><th> Bytes </th><td> 3.20 GB </td> <td> 8.00 MB </td></tr>\n",
       "    <tr><th> Shape </th><td> (20000, 20000) </td> <td> (1000, 1000) </td></tr>\n",
       "    <tr><th> Count </th><td> 400 Tasks </td><td> 400 Chunks </td></tr>\n",
       "    <tr><th> Type </th><td> float64 </td><td> numpy.ndarray </td></tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</td>\n",
       "<td>\n",
       "<svg width=\"170\" height=\"170\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"120\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"0\" y1=\"6\" x2=\"120\" y2=\"6\" />\n",
       "  <line x1=\"0\" y1=\"12\" x2=\"120\" y2=\"12\" />\n",
       "  <line x1=\"0\" y1=\"18\" x2=\"120\" y2=\"18\" />\n",
       "  <line x1=\"0\" y1=\"24\" x2=\"120\" y2=\"24\" />\n",
       "  <line x1=\"0\" y1=\"30\" x2=\"120\" y2=\"30\" />\n",
       "  <line x1=\"0\" y1=\"36\" x2=\"120\" y2=\"36\" />\n",
       "  <line x1=\"0\" y1=\"42\" x2=\"120\" y2=\"42\" />\n",
       "  <line x1=\"0\" y1=\"48\" x2=\"120\" y2=\"48\" />\n",
       "  <line x1=\"0\" y1=\"54\" x2=\"120\" y2=\"54\" />\n",
       "  <line x1=\"0\" y1=\"60\" x2=\"120\" y2=\"60\" />\n",
       "  <line x1=\"0\" y1=\"66\" x2=\"120\" y2=\"66\" />\n",
       "  <line x1=\"0\" y1=\"72\" x2=\"120\" y2=\"72\" />\n",
       "  <line x1=\"0\" y1=\"78\" x2=\"120\" y2=\"78\" />\n",
       "  <line x1=\"0\" y1=\"84\" x2=\"120\" y2=\"84\" />\n",
       "  <line x1=\"0\" y1=\"90\" x2=\"120\" y2=\"90\" />\n",
       "  <line x1=\"0\" y1=\"96\" x2=\"120\" y2=\"96\" />\n",
       "  <line x1=\"0\" y1=\"102\" x2=\"120\" y2=\"102\" />\n",
       "  <line x1=\"0\" y1=\"108\" x2=\"120\" y2=\"108\" />\n",
       "  <line x1=\"0\" y1=\"120\" x2=\"120\" y2=\"120\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"0\" y2=\"120\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"6\" y1=\"0\" x2=\"6\" y2=\"120\" />\n",
       "  <line x1=\"12\" y1=\"0\" x2=\"12\" y2=\"120\" />\n",
       "  <line x1=\"18\" y1=\"0\" x2=\"18\" y2=\"120\" />\n",
       "  <line x1=\"24\" y1=\"0\" x2=\"24\" y2=\"120\" />\n",
       "  <line x1=\"30\" y1=\"0\" x2=\"30\" y2=\"120\" />\n",
       "  <line x1=\"36\" y1=\"0\" x2=\"36\" y2=\"120\" />\n",
       "  <line x1=\"42\" y1=\"0\" x2=\"42\" y2=\"120\" />\n",
       "  <line x1=\"48\" y1=\"0\" x2=\"48\" y2=\"120\" />\n",
       "  <line x1=\"54\" y1=\"0\" x2=\"54\" y2=\"120\" />\n",
       "  <line x1=\"60\" y1=\"0\" x2=\"60\" y2=\"120\" />\n",
       "  <line x1=\"66\" y1=\"0\" x2=\"66\" y2=\"120\" />\n",
       "  <line x1=\"72\" y1=\"0\" x2=\"72\" y2=\"120\" />\n",
       "  <line x1=\"78\" y1=\"0\" x2=\"78\" y2=\"120\" />\n",
       "  <line x1=\"84\" y1=\"0\" x2=\"84\" y2=\"120\" />\n",
       "  <line x1=\"90\" y1=\"0\" x2=\"90\" y2=\"120\" />\n",
       "  <line x1=\"96\" y1=\"0\" x2=\"96\" y2=\"120\" />\n",
       "  <line x1=\"102\" y1=\"0\" x2=\"102\" y2=\"120\" />\n",
       "  <line x1=\"108\" y1=\"0\" x2=\"108\" y2=\"120\" />\n",
       "  <line x1=\"120\" y1=\"0\" x2=\"120\" y2=\"120\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"0.0,0.0 120.0,0.0 120.0,120.0 0.0,120.0\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"60.000000\" y=\"140.000000\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >20000</text>\n",
       "  <text x=\"140.000000\" y=\"60.000000\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,140.000000,60.000000)\">20000</text>\n",
       "</svg>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "dask.array<normal, shape=(20000, 20000), dtype=float64, chunksize=(1000, 1000), chunktype=numpy.ndarray>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import dask.array as da\n",
    "\n",
    "x = da.random.normal(10, 0.1, size=(20000, 20000),   # 400 million element array \n",
    "                              chunks=(1000, 1000))   # Cut into 1000x1000 sized chunks\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr>\n",
       "<td>\n",
       "<table>\n",
       "  <thead>\n",
       "    <tr><td> </td><th> Array </th><th> Chunk </th></tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr><th> Bytes </th><td> 1.60 kB </td> <td> 80 B </td></tr>\n",
       "    <tr><th> Shape </th><td> (200,) </td> <td> (10,) </td></tr>\n",
       "    <tr><th> Count </th><td> 980 Tasks </td><td> 20 Chunks </td></tr>\n",
       "    <tr><th> Type </th><td> float64 </td><td> numpy.ndarray </td></tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</td>\n",
       "<td>\n",
       "<svg width=\"170\" height=\"75\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"120\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"0\" y1=\"25\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"0\" y2=\"25\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"6\" y1=\"0\" x2=\"6\" y2=\"25\" />\n",
       "  <line x1=\"12\" y1=\"0\" x2=\"12\" y2=\"25\" />\n",
       "  <line x1=\"18\" y1=\"0\" x2=\"18\" y2=\"25\" />\n",
       "  <line x1=\"24\" y1=\"0\" x2=\"24\" y2=\"25\" />\n",
       "  <line x1=\"30\" y1=\"0\" x2=\"30\" y2=\"25\" />\n",
       "  <line x1=\"36\" y1=\"0\" x2=\"36\" y2=\"25\" />\n",
       "  <line x1=\"42\" y1=\"0\" x2=\"42\" y2=\"25\" />\n",
       "  <line x1=\"48\" y1=\"0\" x2=\"48\" y2=\"25\" />\n",
       "  <line x1=\"54\" y1=\"0\" x2=\"54\" y2=\"25\" />\n",
       "  <line x1=\"60\" y1=\"0\" x2=\"60\" y2=\"25\" />\n",
       "  <line x1=\"66\" y1=\"0\" x2=\"66\" y2=\"25\" />\n",
       "  <line x1=\"72\" y1=\"0\" x2=\"72\" y2=\"25\" />\n",
       "  <line x1=\"78\" y1=\"0\" x2=\"78\" y2=\"25\" />\n",
       "  <line x1=\"84\" y1=\"0\" x2=\"84\" y2=\"25\" />\n",
       "  <line x1=\"90\" y1=\"0\" x2=\"90\" y2=\"25\" />\n",
       "  <line x1=\"96\" y1=\"0\" x2=\"96\" y2=\"25\" />\n",
       "  <line x1=\"102\" y1=\"0\" x2=\"102\" y2=\"25\" />\n",
       "  <line x1=\"108\" y1=\"0\" x2=\"108\" y2=\"25\" />\n",
       "  <line x1=\"120\" y1=\"0\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"0.0,0.0 120.0,0.0 120.0,25.412616514582485 0.0,25.412616514582485\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"60.000000\" y=\"45.412617\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >200</text>\n",
       "  <text x=\"140.000000\" y=\"12.706308\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(0,140.000000,12.706308)\">1</text>\n",
       "</svg>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "dask.array<getitem, shape=(200,), dtype=float64, chunksize=(10,), chunktype=numpy.ndarray>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = x.mean(axis=0)[::100]                            # Perform NumPy-style operations\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.56 s, sys: 407 ms, total: 1.96 s\n",
      "Wall time: 5.31 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([10.00001267, 10.00069835,  9.99915538, 10.0007113 ,  9.99959944,\n",
       "        9.99988025, 10.00031735, 10.00004321,  9.99983228, 10.00035415,\n",
       "        9.99969681,  9.99919569, 10.00032529,  9.99951196,  9.99989431,\n",
       "        9.99935589,  9.99988345,  9.99995694, 10.00102528, 10.0009544 ,\n",
       "        9.9996422 ,  9.99943033,  9.99970901, 10.00089393, 10.00041657,\n",
       "       10.00009736,  9.99944552, 10.00064095,  9.9990806 ,  9.99996886,\n",
       "       10.00025507, 10.00002207,  9.99995079,  9.99940929, 10.00014614,\n",
       "        9.99945231,  9.99830028,  9.99958802,  9.99925128, 10.00054042,\n",
       "       10.00111489,  9.99906685,  9.99993243, 10.00154676,  9.99909141,\n",
       "       10.00013003, 10.00036337,  9.99939101, 10.00100796,  9.99989416,\n",
       "       10.00069069,  9.99975335, 10.00009134, 10.00083246,  9.9996698 ,\n",
       "        9.99958651, 10.00040494, 10.00092728,  9.99901134, 10.00016869,\n",
       "       10.00035366,  9.99902707,  9.99997935,  9.9996051 ,  9.9994696 ,\n",
       "       10.00054379,  9.99941003,  9.99948602, 10.0013534 , 10.00005083,\n",
       "       10.00010287, 10.00037773,  9.999551  , 10.00168349,  9.99960294,\n",
       "        9.99989122, 10.00018859,  9.99959029,  9.9996496 ,  9.99873958,\n",
       "       10.00102741,  9.9987192 , 10.00017428, 10.00071711,  9.99931866,\n",
       "        9.99992714, 10.00056532, 10.00014321,  9.99955516, 10.00032495,\n",
       "       10.00043897,  9.99938956, 10.00017438,  9.99926863,  9.99895143,\n",
       "       10.00018819,  9.999362  , 10.00004216, 10.00063798,  9.99961699,\n",
       "       10.00134904,  9.99987302, 10.00055964, 10.00056317,  9.99970845,\n",
       "       10.00005016, 10.00080093, 10.00083108,  9.99973802, 10.00062057,\n",
       "       10.00082677,  9.99943406, 10.00113362,  9.99928087,  9.99959594,\n",
       "       10.00232912,  9.99913823,  9.99984353, 10.00040551, 10.00135492,\n",
       "       10.00079043,  9.99904655, 10.00120892,  9.99917757,  9.99914248,\n",
       "       10.00084851, 10.00056055,  9.99878261,  9.9995246 , 10.0007825 ,\n",
       "        9.99942868, 10.00069814, 10.00005767,  9.9997358 , 10.00080368,\n",
       "       10.00065294,  9.9993224 , 10.00075108, 10.00099671, 10.00015609,\n",
       "       10.00059338,  9.99991257,  9.99864698, 10.0001088 , 10.00041877,\n",
       "       10.00022327, 10.00059846,  9.99878039, 10.00007926, 10.00131069,\n",
       "        9.99998762, 10.00097488, 10.00089013, 10.00090509, 10.00004973,\n",
       "        9.99925873, 10.00102462, 10.0008518 ,  9.99871489,  9.99992086,\n",
       "       10.00100224, 10.0000285 , 10.00075364,  9.99960623, 10.00097673,\n",
       "        9.99916358, 10.00055457,  9.99969487, 10.00096325,  9.99877996,\n",
       "       10.0008049 ,  9.99957052, 10.00008434, 10.00128024,  9.99937786,\n",
       "       10.00023375,  9.99854692,  9.99953375,  9.9992804 , 10.00035883,\n",
       "       10.00069415, 10.00057084, 10.00033846,  9.99973005, 10.0005851 ,\n",
       "       10.00031733,  9.99991468, 10.00083673, 10.00059471, 10.00120084,\n",
       "       10.00022066, 10.00008023, 10.00050418,  9.99929553, 10.00031873,\n",
       "       10.00075279,  9.99858455, 10.00087787,  9.99932857,  9.99949481])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "y.compute()     # Time to compute the result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Performance comparison\n",
    "---------------------------\n",
    "\n",
    "The following experiment was performed on a heavy personal laptop.  Your performance may vary.  If you attempt the NumPy version then please ensure that you have more than 4GB of main memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NumPy: 19s, Needs gigabytes of memory**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import numpy as np\n",
    "\n",
    "%%time \n",
    "x = np.random.normal(10, 0.1, size=(20000, 20000)) \n",
    "y = x.mean(axis=0)[::100] \n",
    "y\n",
    "\n",
    "CPU times: user 19.6 s, sys: 160 ms, total: 19.8 s\n",
    "Wall time: 19.7 s\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dask Array: 4s, Needs megabytes of memory**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import dask.array as da\n",
    "\n",
    "%%time\n",
    "x = da.random.normal(10, 0.1, size=(20000, 20000), chunks=(1000, 1000))\n",
    "y = x.mean(axis=0)[::100] \n",
    "y.compute() \n",
    "\n",
    "CPU times: user 29.4 s, sys: 1.07 s, total: 30.5 s\n",
    "Wall time: 4.01 s\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the Dask array computation ran in 4 seconds, but used 29.4 seconds of user CPU time. The numpy computation ran in 19.7 seconds and used 19.6 seconds of user CPU time.\n",
    "\n",
    "Dask finished faster, but used more total CPU time because Dask was able to transparently parallelize the computation because of the chunk size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Questions*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*  What happens if the dask chunks=(20000,20000)?\n",
    "    * Will the computation run in 4 seconds?\n",
    "    * How much memory will be used?\n",
    "* What happens if the dask chunks=(25,25)?\n",
    "    * What happens to CPU and memory?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Limitations\n",
    "-----------\n",
    "\n",
    "Dask Array does not implement the entire numpy interface.  Users expecting this\n",
    "will be disappointed.  Notably Dask Array has the following failings:\n",
    "\n",
    "1.  Dask does not implement all of ``np.linalg``.  This has been done by a\n",
    "    number of excellent BLAS/LAPACK implementations and is the focus of\n",
    "    numerous ongoing academic research projects.\n",
    "2.  Dask Array does not support some operations where the resulting shape\n",
    "    depends on the values of the array. For those that it does support\n",
    "    (for example, masking one Dask Array with another boolean mask),\n",
    "    the chunk sizes will be unknown, which may cause issues with other\n",
    "    operations that need to know the chunk sizes.\n",
    "3.  Dask Array does not attempt operations like ``sort`` which are notoriously\n",
    "    difficult to do in parallel and are of somewhat diminished value on very\n",
    "    large data (you rarely actually need a full sort).\n",
    "    Often we include parallel-friendly alternatives like ``topk``.\n",
    "4.  Dask development is driven by immediate need, and so many lesser used\n",
    "    functions, like ``np.sometrue`` have not been implemented purely out of\n",
    "    laziness.  These would make excellent community contributions.\n",
    "    \n",
    "* [Array documentation](https://docs.dask.org/en/latest/array.html)\n",
    "* [Array screencast](https://youtu.be/9h_61hXCDuI)\n",
    "* [Array API](https://docs.dask.org/en/latest/array-api.html)\n",
    "* [Array examples](https://examples.dask.org/array.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dask DataFrames\n",
    "\n",
    "Pandas is great for tabular datasets that fit in memory. Dask becomes useful when the dataset you want to analyze is larger than your machine's RAM. The demo dataset we're working with is only about 3MB, but `dask.dataframe` will scale to  datasets much larger than memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://examples.dask.org/_images/dask-dataframe.svg\" align=\"right\" width=\"28%\">\n",
    "\n",
    "The `dask.dataframe` module implements a blocked parallel `DataFrame` object that mimics a large subset of the Pandas `DataFrame` API. One Dask `DataFrame` is comprised of many in-memory pandas `DataFrames` separated along the index. One operation on a Dask `DataFrame` triggers many pandas operations on the constituent pandas `DataFrame`s in a way that is mindful of potential parallelism and memory constraints.\n",
    "\n",
    "**Related Documentation**\n",
    "\n",
    "* [DataFrame documentation](https://docs.dask.org/en/latest/dataframe.html)\n",
    "* [DataFrame screencast](https://youtu.be/AT2XtFehFSQ)\n",
    "* [DataFrame API](https://docs.dask.org/en/latest/dataframe-api.html)\n",
    "* [DataFrame examples](https://examples.dask.org/dataframe.html)\n",
    "* [Pandas documentation](https://pandas.pydata.org/pandas-docs/stable/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's convert our dask bag into a dataframe for more efficient processing. We'll first have to normalize some of the nested structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><strong>Dask DataFrame Structure:</strong></div>\n",
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>name</th>\n",
       "      <th>occupation</th>\n",
       "      <th>telephone</th>\n",
       "      <th>address</th>\n",
       "      <th>credit-card</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>npartitions=10</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>int64</td>\n",
       "      <td>object</td>\n",
       "      <td>object</td>\n",
       "      <td>object</td>\n",
       "      <td>object</td>\n",
       "      <td>object</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "<div>Dask Name: from-delayed, 20 tasks</div>"
      ],
      "text/plain": [
       "Dask DataFrame Structure:\n",
       "                  age    name occupation telephone address credit-card\n",
       "npartitions=10                                                        \n",
       "                int64  object     object    object  object      object\n",
       "                  ...     ...        ...       ...     ...         ...\n",
       "...               ...     ...        ...       ...     ...         ...\n",
       "                  ...     ...        ...       ...     ...         ...\n",
       "                  ...     ...        ...       ...     ...         ...\n",
       "Dask Name: from-delayed, 20 tasks"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import dask.dataframe as dd\n",
    "\n",
    "df = dd.read_json(\n",
    "    \"data/*.json\",\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can operate on the dataframe as if it is a pandas object. Note that some of our items are `object` type because the records were not normalized. We can normalize them using `df.apply`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><strong>Dask DataFrame Structure:</strong></div>\n",
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>name</th>\n",
       "      <th>occupation</th>\n",
       "      <th>telephone</th>\n",
       "      <th>address</th>\n",
       "      <th>credit-card</th>\n",
       "      <th>city</th>\n",
       "      <th>fullname</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>npartitions=10</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>int64</td>\n",
       "      <td>object</td>\n",
       "      <td>object</td>\n",
       "      <td>object</td>\n",
       "      <td>object</td>\n",
       "      <td>object</td>\n",
       "      <td>object</td>\n",
       "      <td>object</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "<div>Dask Name: assign, 80 tasks</div>"
      ],
      "text/plain": [
       "Dask DataFrame Structure:\n",
       "                  age    name occupation telephone address credit-card    city fullname\n",
       "npartitions=10                                                                         \n",
       "                int64  object     object    object  object      object  object   object\n",
       "                  ...     ...        ...       ...     ...         ...     ...      ...\n",
       "...               ...     ...        ...       ...     ...         ...     ...      ...\n",
       "                  ...     ...        ...       ...     ...         ...     ...      ...\n",
       "                  ...     ...        ...       ...     ...         ...     ...      ...\n",
       "Dask Name: assign, 80 tasks"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"city\"] = df.address.apply(lambda rec: rec[\"city\"], meta=(\"city\", str))\n",
    "df[\"fullname\"] = df.name.apply(lambda l: \" \".join(l), meta=(\"fullname\", str))\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>name</th>\n",
       "      <th>occupation</th>\n",
       "      <th>telephone</th>\n",
       "      <th>address</th>\n",
       "      <th>credit-card</th>\n",
       "      <th>city</th>\n",
       "      <th>fullname</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>66</td>\n",
       "      <td>[Lael, Randall]</td>\n",
       "      <td>Breeder</td>\n",
       "      <td>591-017-9956</td>\n",
       "      <td>{'address': '390 Saint Josephs Path', 'city': ...</td>\n",
       "      <td>{'number': '4314 8194 2212 8203', 'expiration-...</td>\n",
       "      <td>Williamsport</td>\n",
       "      <td>Lael Randall</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>66</td>\n",
       "      <td>[Kamilah, Moss]</td>\n",
       "      <td>Artexer</td>\n",
       "      <td>955.665.1266</td>\n",
       "      <td>{'address': '736 Dearborn Concession road', 'c...</td>\n",
       "      <td>{'number': '2343 6039 9694 2796', 'expiration-...</td>\n",
       "      <td>Cleburne</td>\n",
       "      <td>Kamilah Moss</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>66</td>\n",
       "      <td>[Bill, Robinson]</td>\n",
       "      <td>Hod Carrier</td>\n",
       "      <td>255.137.8893</td>\n",
       "      <td>{'address': '1032 Berry Extension Plaza', 'cit...</td>\n",
       "      <td>{'number': '4651 9150 0379 3075', 'expiration-...</td>\n",
       "      <td>Temple Terrace</td>\n",
       "      <td>Bill Robinson</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>66</td>\n",
       "      <td>[Douglas, Reid]</td>\n",
       "      <td>Golf Club Professional</td>\n",
       "      <td>(264) 564-4088</td>\n",
       "      <td>{'address': '357 Arco Glen', 'city': 'Harker H...</td>\n",
       "      <td>{'number': '4196 4194 6405 7866', 'expiration-...</td>\n",
       "      <td>Harker Heights</td>\n",
       "      <td>Douglas Reid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>66</td>\n",
       "      <td>[January, Wilson]</td>\n",
       "      <td>Employee</td>\n",
       "      <td>751.109.4791</td>\n",
       "      <td>{'address': '58 Cambon Mews', 'city': 'Ankeny'}</td>\n",
       "      <td>{'number': '4620 3728 0324 4651', 'expiration-...</td>\n",
       "      <td>Ankeny</td>\n",
       "      <td>January Wilson</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     age               name              occupation       telephone  \\\n",
       "17    66    [Lael, Randall]                 Breeder    591-017-9956   \n",
       "46    66    [Kamilah, Moss]                 Artexer    955.665.1266   \n",
       "47    66   [Bill, Robinson]             Hod Carrier    255.137.8893   \n",
       "75    66    [Douglas, Reid]  Golf Club Professional  (264) 564-4088   \n",
       "143   66  [January, Wilson]                Employee    751.109.4791   \n",
       "\n",
       "                                               address  \\\n",
       "17   {'address': '390 Saint Josephs Path', 'city': ...   \n",
       "46   {'address': '736 Dearborn Concession road', 'c...   \n",
       "47   {'address': '1032 Berry Extension Plaza', 'cit...   \n",
       "75   {'address': '357 Arco Glen', 'city': 'Harker H...   \n",
       "143    {'address': '58 Cambon Mews', 'city': 'Ankeny'}   \n",
       "\n",
       "                                           credit-card            city  \\\n",
       "17   {'number': '4314 8194 2212 8203', 'expiration-...    Williamsport   \n",
       "46   {'number': '2343 6039 9694 2796', 'expiration-...        Cleburne   \n",
       "47   {'number': '4651 9150 0379 3075', 'expiration-...  Temple Terrace   \n",
       "75   {'number': '4196 4194 6405 7866', 'expiration-...  Harker Heights   \n",
       "143  {'number': '4620 3728 0324 4651', 'expiration-...          Ankeny   \n",
       "\n",
       "           fullname  \n",
       "17     Lael Randall  \n",
       "46     Kamilah Moss  \n",
       "47    Bill Robinson  \n",
       "75     Douglas Reid  \n",
       "143  January Wilson  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.nlargest(columns=\"age\").compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>occupation</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Accountant</th>\n",
       "      <td>48.888889</td>\n",
       "      <td>North Platte</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Accounts Assistant</th>\n",
       "      <td>42.000000</td>\n",
       "      <td>Eagle Mountain</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Accounts Clerk</th>\n",
       "      <td>45.857143</td>\n",
       "      <td>Amsterdam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Accounts Manager</th>\n",
       "      <td>40.000000</td>\n",
       "      <td>Gloucester</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Accounts Staff</th>\n",
       "      <td>38.666667</td>\n",
       "      <td>Hallandale Beach</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Youth Worker</th>\n",
       "      <td>37.857143</td>\n",
       "      <td>Chesterfield</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Zoo Keeper</th>\n",
       "      <td>42.800000</td>\n",
       "      <td>Upland</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Zoo Manager</th>\n",
       "      <td>42.916667</td>\n",
       "      <td>Goodlettsville</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Zoologist</th>\n",
       "      <td>25.750000</td>\n",
       "      <td>Lorain</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tailor</th>\n",
       "      <td>30.000000</td>\n",
       "      <td>Pflugerville</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1156 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          age              city\n",
       "occupation                                     \n",
       "Accountant          48.888889      North Platte\n",
       "Accounts Assistant  42.000000    Eagle Mountain\n",
       "Accounts Clerk      45.857143         Amsterdam\n",
       "Accounts Manager    40.000000        Gloucester\n",
       "Accounts Staff      38.666667  Hallandale Beach\n",
       "...                       ...               ...\n",
       "Youth Worker        37.857143      Chesterfield\n",
       "Zoo Keeper          42.800000            Upland\n",
       "Zoo Manager         42.916667    Goodlettsville\n",
       "Zoologist           25.750000            Lorain\n",
       "Tailor              30.000000      Pflugerville\n",
       "\n",
       "[1156 rows x 2 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"occupation\").aggregate({\"age\": \"mean\", \"city\": \"first\"}).compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "distributed.client - ERROR - Failed to reconnect to scheduler after 10.00 seconds, closing client\n",
      "_GatheringFuture exception was never retrieved\n",
      "future: <_GatheringFuture finished exception=CancelledError()>\n",
      "asyncio.exceptions.CancelledError\n"
     ]
    }
   ],
   "source": [
    "client.shutdown()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}