tutorial/tutorial_v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3tW2MkdrIn6F"
   },
   "source": [

    "\n",
    "## Installation\n",
    "\n",
    "To run Jupyter notebook locally:\n",
    "\n",
    "```\n",
    "python3 -m pip install gcsfs waymo-open-dataset-tf-2-12-0==1.6.4\n",
    "python3 -m pip install \"notebook>=5.3\" \"ipywidgets>=7.5\"\n",
    "python3 -m pip install --upgrade jupyter_http_over_ws>=0.0.7 && \\\n",
    "jupyter serverextension enable --py jupyter_http_over_ws\n",
    "jupyter notebook\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CaVf78MSp3q0"
   },
   "source": [
    "# Overview\n",
    "\n",
    "The Waymo Open Dataset (WOD), a comprehensive self-driving dataset, has recently undergone a major update to its format. The new format is based on the [Apache Parquet column-oriented file format](https://parquet.apache.org/docs/file-format/). This format separates the data into multiple tables, allowing users to selectively download the portion of the dataset needed for their specific use case. This modular format offers a significant advantage over the previous format by reducing the amount of data that needs to be downloaded and processed, saving time and resources.\n",
    "\n",
    "This tutorial explores the benefits of the new format and how to effectively access and work with the selective parts of the WOD. \n",
    "\n",
    "Whether you are an experienced data scientist or just starting out in the field of autonomous driving, this tutorial will provide you with the information and tools you need to prepare a training dataset derived from WOD for your model.\n",
    "\n",
    "## Navigating this tutorial\n",
    "\n",
    "Text and code cells in this notebook use headers and cell titles which form a hierarchy to help navigate this tutorial:\n",
    "\n",
    "- Overview\n",
    "- Usage example:\n",
    "  - Initial setup\n",
    "  - Basic Example\n",
    "  - Joining components for different applications\n",
    "    - Scene or track level\n",
    "      - Per-object trajectories\n",
    "    - Frame level\n",
    "      - Sensor data with both lidar and camera boxes\n",
    "      - Sensor data with lidar and camera boxes if available\n",
    "    - Object level\n",
    "      - Camera image, lidar, boxes with keypoints\n",
    "- Dataset Format\n",
    "    - Folder structure\n",
    "    - Loading raw data\n",
    "    - Supported components\n",
    "    - Object-oriented API\n",
    "    - A relational dataset-like structure\n",
    "\n",
    "\n",
    "Cell titles make sense in the hierarchy, ex. \"Joining components for different applications > Frame level > Sensor data ...\". Use \"Table of contents\" in Google Colab to see.\n",
    "\n",
    "The first section \"Usage example\" has a number of motivating examples, feel free\n",
    "to jump to the \"Dataset Format\" if you'd like to learn about technical details\n",
    "first.\n",
    "\n",
    "## v2 dataset format\n",
    "\n",
    "Previous releases of the WOD were in the Frame binary protocol buffers serialized into tfrecords files. From now on we will refer to this data format as v1. With WOD `v2.0.0.alpha` we are launching the v2 format of the dataset. Note that column values in v2-supported components is the same as corresponding proto fields in the `v1.4.2`. Refer to the section \"List columns in all components\" for the full list of supported components and the code under `waymo_open_dataset/v2/perception/compat_v1/` to learn more how this correspondence is defined.\n",
    "\n",
    "## v2 API\n",
    "\n",
    "The v2 format was designed to be usable with any library which supports reading the Apache Parquet file format, without any extra dependencies (not even protocol buffers). See \"Loading raw data\" section for an example. \n",
    "\n",
    "We provide a collection of convenience functions and dataclasses, the main of which is `v2.merge` - a small wrapper for `DataFrame.merge` which exploits the v2 format conventions to significantly simplify merging multiple components into a single DataFrame object. To learn more refer to the docstring for the `merge` function in `waymo_open_dataset/v2/dataframe_utils.py` and `waymo_open_dataset/v2/__init__.py` to see the full surface of the v2 object oriented API for WOD."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SCnlKDi-nqyU"
   },
   "source": [
    "# Usage examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "S7v6WUnPdOS9"
   },
   "outputs": [],
   "source": [
    "#@title Initial setup\n",
    "from typing import Optional\n",
    "import warnings\n",
    "# Disable annoying warnings from PyArrow using under the hood.\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "\n",
    "\n",
    "import tensorflow as tf\n",
    "import dask.dataframe as dd\n",
    "from waymo_open_dataset import v2\n",
    "\n",
    "\n",
    "# Path to the directory with all components\n",
    "dataset_dir = '<specify actual path>'\n",

    "\n",
    "context_name = '10023947602400723454_1120_000_1140_000'\n",
    "\n",
    "def read(tag: str) -> dd.DataFrame:\n",
    "  \"\"\"Creates a Dask DataFrame for the component specified by its tag.\"\"\"\n",
    "  paths = tf.io.gfile.glob(f'{dataset_dir}/{tag}/{context_name}.parquet')\n",
    "  return dd.read_parquet(paths)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "mxF3gfoEgKhs",
    "outputId": "2b486f2c-7def-47bf-ea05-80e02cddc8db"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available 979 rows:\n",
      "context_name: 10023947602400723454_1120_000_1140_000 ts: 1552440195362591 camera_name: 1 image size: 387998 bytes. Has 50 camera labels:\n",
      "\tid: 02f3a769-021f-49b0-b954-18e4fa1d5fde,  center: (900.0, 665.7) px\n",
      "\tid: 0611ea50-f652-406d-942a-9ec7e3e90546,  center: (1303.6, 697.9) px\n",
      "\tid: 08b1e19e-b912-4963-a7bb-55e138bbc25e,  center: (1828.5, 741.5) px\n",
      "\tid: 11165fc3-e936-4722-9257-1574c6752eab,  center: (951.2, 658.7) px\n",
      "\t...\n",
      "context_name: 10023947602400723454_1120_000_1140_000 ts: 1552440195362591 camera_name: 2 image size: 412300 bytes. Has 11 camera labels:\n",
      "\tid: 0ec04ccf-c924-45b1-ab54-cbc82a742877,  center: (703.9, 1108.3) px\n",
      "\tid: 144c7855-9e09-4bda-bb7a-c52a06d3075f,  center: (953.7, 628.4) px\n",
      "\tid: 17742b3c-df43-42a6-ba81-c67474d0990b,  center: (1052.9, 665.7) px\n",
      "\tid: 648739d1-b4d9-467a-8274-a68cbe294d48,  center: (1479.0, 1052.6) px\n",
      "\t...\n",
      "context_name: 10023947602400723454_1120_000_1140_000 ts: 1552440195362591 camera_name: 4 image size: 193053 bytes. Has 1 camera labels:\n",
      "\tid: c6d13c46-9523-4bc6-b445-86afc3bcac2d,  center: (960.0, 693.7) px\n",
      "context_name: 10023947602400723454_1120_000_1140_000 ts: 1552440195362591 camera_name: 3 image size: 431658 bytes. Has 5 camera labels:\n",
      "\tid: 42d3fd29-cc31-44c6-a044-7af1a172488e,  center: (319.6, 747.2) px\n",
      "\tid: 4689e83b-684d-4d3c-a055-0f6215fd82d0,  center: (796.4, 955.0) px\n",
      "\tid: 7020b281-d93a-4256-b71b-f83cc447b463,  center: (337.3, 1030.5) px\n",
      "\tid: 960d036e-c7ed-4e64-b069-86e114574d77,  center: (24.6, 735.2) px\n",
      "\t...\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "# @title Basic Example (Camera images with labels)\n",
    "\n",
    "# Lazily read camera images and boxes \n",
    "cam_image_df = read('camera_image')\n",
    "cam_box_df = read('camera_box')\n",
    "\n",
    "# Combine DataFrame for individual components into a single DataFrame.\n",
    "\n",
    "# Camera cam_box_df will be grouped, so each row will have a camera image\n",
    "# and all associated boxes.\n",
    "image_w_box_df = v2.merge(cam_image_df, cam_box_df, right_group=True)\n",
    "\n",
    "# Show raw data\n",
    "image_w_box_df.head()\n",
    "\n",
    "# Example how to access data fields via v2 object-oriented API\n",
    "print(f'Available {image_w_box_df.shape[0].compute()} rows:')\n",
    "for i, (_, r) in enumerate(image_w_box_df.iterrows()):\n",
    "  # Create component dataclasses for the raw data\n",
    "  cam_image = v2.CameraImageComponent.from_dict(r)\n",
    "  cam_box = v2.CameraBoxComponent.from_dict(r)\n",
    "  print(\n",
    "      f'context_name: {cam_image.key.segment_context_name}'\n",
    "      f' ts: {cam_image.key.frame_timestamp_micros}'\n",
    "      f' camera_name: {cam_image.key.camera_name}'\n",
    "      f' image size: {len(cam_image.image)} bytes.'\n",
    "      f' Has {len(cam_box.key.camera_object_id)} camera labels:'\n",
    "  )\n",
    "\n",
    "  for j, (object_id, x, y) in enumerate(zip(\n",
    "      cam_box.key.camera_object_id, cam_box.box.center.x, cam_box.box.center.y\n",
    "  )):\n",
    "    print(f'\\tid: {object_id},  center: ({x:.1f}, {y:.1f}) px')\n",
    "    if j > 2:\n",
    "      print('\\t...')\n",
    "      break\n",
    "  if i > 2:\n",
    "    print('...')\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Itm65DeIyhxm"
   },
   "source": [
    "## Joining components for different applications\n",
    "\n",
    "Different models often require custom data preparation pipelines because each model has unique data requirements that need to be met for optimal performance. For example, one model may require lidar sensor data to be pre-processed to remove noise or keep only fixed number of points, while another model may require camera images to be processed for object detection. \n",
    "\n",
    "This section shows how to prepare input data for preprocessing pipelines operating at different levels: entire run segment, scene, frame, camera or a crop for a specific object.\n",
    "\n",
    "NOTE: Depending on the level or type of the JOIN operation used to define a table, a single row of the table may contain repeated values for some parts of the data, referring the same original record (ex. repeated camera images or lidar data for each object). See examples below for details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uT_Gr0WNDXP_"
   },
   "source": [
    "### Scene or track level\n",
    "\n",
    "At this level each row of a table represents data for a single object across all frames in a scene."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "EwPs0p5MDgh_",
    "outputId": "2cbf4d5f-1347-4f64-fa57-69d530e97ba9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Object lidar_box.key.laser_object_id='-ItvfksmEcYtVEcOjjRESg' seen on 90 frames\n",
      "Its trajectory accross frames:\n",
      "\tlidar_box.box.center.x=[23.267635429570873, 23.249028348451247, 23.218535695019455, 23.141356138959964, 23.041015064622115, 22.943379720318262, 22.837007592977898, 22.699640942439146, 22.55271614263438, 22.406182537455607, 22.255760798618212, 22.104388865565852, 21.926845604939444, 21.74289930834675, 21.541009551732714, 21.31035498363417, 21.068154290935126, 20.826015581797037, 20.586888317488047, 20.34230145818219, 20.08004165622333, 19.81921650300501, 19.564810605319508, 19.316562933598107, 19.065521244701813, 18.822356058548394, 18.593808704348703, 18.38374361586102, 18.177806150901233, 18.000751103629227, 17.84484811807488, 17.69976946847237, 17.564665781314034, 17.441918124226504, 17.338055039508617, 17.242434248302743, 17.160077564167295, 17.092516316946785, 17.039343006006675, 16.998662457359387, 16.976064796856008, 16.966728632773084, 16.965259847673224, 16.98372797911179, 17.01152510382053, 17.046049620525082, 17.086749356307337, 17.124988964395016, 17.16220335969774, 17.200703987855377, 17.24230028358761, 17.284421082376866, 17.325798877864145, 17.36597811345564, 17.40445371642818, 17.465868675188176, 17.526958722562995, 17.588083154121705, 17.64997823520207, 17.71159213946885, 17.77282130713138, 17.832271292138103, 17.891777437931523, 17.947069454485245, 18.002243724889922, 18.05748569836942, 18.11273002974849, 18.147924935432457, 18.183069950031495, 18.21832549624196, 18.253590833857743, 18.28874111713776, 18.323933299659984, 18.358708210669647, 18.393407766325254, 18.428071064092364, 18.464965033595945, 18.501744363866237, 18.538620734594588, 18.575460965093953, 18.612194072855345, 18.649138887105437, 18.685815861988885, 18.721472344630456, 18.761109284745544, 18.800532794915853, 18.839777388606308, 18.87910592530352, 18.91836300217983, 18.957596618454772]\n",
      "\tlidar_box.box.center.y=[39.67021106277571, 38.83769751114778, 38.009806525091335, 37.20762452279814, 36.41750437486189, 35.622985224859804, 34.82723138180336, 34.048279891754646, 33.2766528112661, 32.507130800269806, 31.739261217144303, 31.01591207185629, 30.30796640893095, 29.60406351792699, 28.913442462213425, 28.27356134601905, 27.643660114832528, 27.015418976399815, 26.387012054570732, 25.766622460816507, 25.164367119810777, 24.56810236426645, 23.97624993959107, 23.38763555568221, 22.80956746475931, 22.231941601045037, 21.650117758470515, 21.060221458823435, 20.47414905250116, 19.881345395671815, 19.27837801171154, 18.673612431586662, 18.06582241659271, 17.453685910199056, 16.831373808869103, 16.20852074162576, 15.582662673446976, 14.931319845518374, 14.277879116167242, 13.619934974662101, 12.913370249809304, 12.199816191852733, 11.484760584873584, 10.84789209021801, 10.144457385751593, 9.438554552354617, 8.730202467930212, 8.022535077101566, 7.276668992255509, 6.531461873384615, 5.7868379252049635, 5.041601404365792, 4.296417716655924, 3.5513409078303084, 2.8063973488133342, 2.015496123551202, 1.2243735498268506, 0.43319240151504346, -0.3581823762906424, -1.1490338230978523, -1.939777875555592, -2.7394082893697487, -3.539221514854944, -4.3780282016523415, -5.217406577001384, -6.056661249393073, -6.895877720855424, -7.769598043194492, -8.642968708780245, -9.516352046628526, -10.390092801437277, -11.263603989917101, -12.13690737809884, -13.035387556388741, -13.934123088836714, -14.832340250341076, -15.752635952991113, -16.672242258780898, -17.592403078118878, -18.51204755542858, -19.43196839380289, -20.351646529014033, -21.27187145512653, -22.21139014367327, -23.147429253005612, -24.12011640011042, -25.093026782276866, -26.065915373932512, -27.03882279003119, -28.011413585238188]\n",
      "\tlidar_box.box.center.z=[2.4336090585243255, 2.4601563783312486, 2.46606704837086, 2.448118980875279, 2.415364733751858, 2.4094492993400536, 2.4603608242135806, 2.5103296442798637, 2.526728237562793, 2.4721694114604134, 2.415270621982188, 2.389142056724836, 2.374562061748975, 2.358168159058323, 2.3381961105664004, 2.337437441802763, 2.323788068378917, 2.3233269612477443, 2.3188265823318943, 2.3118005292315047, 2.3179060040544073, 2.3159365870031934, 2.30580008871641, 2.2747274539366344, 2.250374766247205, 2.227060928346191, 2.205769465150638, 2.194585978914006, 2.182596061634058, 2.1664154794408432, 2.1396618427590965, 2.117143882773007, 2.100538922033138, 2.096826624292248, 2.0911981332438927, 2.09489927071661, 2.0854502026362525, 2.07660287337751, 2.0642016098240674, 2.0507405936505165, 2.0414193146280013, 2.0353583472287937, 2.029039632188187, 2.0189294250960756, 2.004779307378044, 1.996563172870765, 1.9834723640175298, 1.9627735859390896, 1.943477639766627, 1.935285769579309, 1.924604152030895, 1.9080634992299679, 1.8959641603279351, 1.8893467945788416, 1.8783260990711028, 1.866494677778519, 1.8582929581402823, 1.8467840965904543, 1.8334302364617656, 1.8218337018590205, 1.8100224462430106, 1.7881794907457902, 1.768047063262543, 1.7450815517340743, 1.7247015639946426, 1.7047411142455644, 1.6847815867525924, 1.6742237097567454, 1.6612749062386456, 1.6492362881103872, 1.6336215379519103, 1.6195897225337603, 1.6065549588190606, 1.5943873432336204, 1.5810543671130972, 1.5694965486234196, 1.5543435368254563, 1.5406203089547716, 1.5268890501725423, 1.5131654462338702, 1.5012225261081085, 1.4857135632630047, 1.475810241132308, 1.4579540534050466, 1.4457922453239576, 1.429479122565965, 1.4149615281365868, 1.4004445353886013, 1.385927026304671, 1.3687306483608381]\n"
     ]
    }
   ],
   "source": [
    "# @title Per-object trajectories\n",
    "\n",
    "# Lazily read lidar boxes\n",
    "lidar_box_df = read('lidar_box')\n",
    "# Group all available boxes per object\n",
    "lidar_box_df = (\n",
    "    lidar_box_df.groupby(['key.segment_context_name', 'key.laser_object_id'])\n",
    "    .agg(list)\n",
    "    .reset_index()\n",
    ")\n",
    "\n",
    "\n",
    "# Read a single row, which contains data for all available frames.\n",
    "_, row = next(iter(lidar_box_df.iterrows()))\n",
    "\n",
    "# Create component object\n",
    "lidar_box = v2.LiDARBoxComponent.from_dict(row)\n",
    "print(\n",
    "    f'Object {lidar_box.key.laser_object_id=} seen on'\n",
    "    f' {len(lidar_box.key.frame_timestamp_micros)} frames'\n",
    ")\n",
    "print('Its trajectory across frames:')\n",
    "print(f'\\t{lidar_box.box.center.x=}')\n",
    "print(f'\\t{lidar_box.box.center.y=}')\n",
    "print(f'\\t{lidar_box.box.center.z=}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c3JEPh7XDMVW"
   },
   "source": [
    "### Frame level\n",
    "\n",
    "At this level each row of a table represents a single frame with all objects in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "4zH_DeBedGEQ",
    "outputId": "4bb32cf7-fb5a-46db-91f3-2456cc5bbcae"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 24 objects on lidar.key.segment_context_name='10023947602400723454_1120_000_1140_000' lidar.key.frame_timestamp_micros=1552440195362591\n",
      "\tlaser_object_id='ZyK_iICxQsEYdJLxMFTw7w' camera_object_id='08b1e19e-b912-4963-a7bb-55e138bbc25e' camera_name=1\n",
      "\tlaser_object_id='ZyK_iICxQsEYdJLxMFTw7w' camera_object_id='da518036-17cb-48c4-97aa-97939f989d16' camera_name=3\n",
      "\tlaser_object_id='dNW3S4yA8s8GmltmAPV8LQ' camera_object_id='24508c28-3154-4b7f-964d-70b3ed7d5a9d' camera_name=1\n",
      "\tlaser_object_id='8IZ7fkXm0FDeUdEuJAqSlA' camera_object_id='260f57c0-787d-4f36-bfe9-64ca9828448f' camera_name=1\n",
      "\tlaser_object_id='brLOJzgVMhP_-kdQzIa7ng' camera_object_id='30326915-a157-4c60-9bef-0a188938c998' camera_name=1\n",
      "\tlaser_object_id='txvVqORJ6Gcf510DdJvW4Q' camera_object_id='4286a233-4a34-4d99-b214-2d8690e92570' camera_name=1\n",
      "\tlaser_object_id='L5HBM5tSKtNA4qHegDQk8Q' camera_object_id='4b9a9206-fbf1-4d45-8765-197759100bd6' camera_name=1\n",
      "\tlaser_object_id='YTw2k0_-8UqolJn4WQRc2g' camera_object_id='699b6cb3-afc1-469f-8ac9-40978aeb511b' camera_name=1\n",
      "\tlaser_object_id='Yyu039jUMIJ4gI_2-mTkSg' camera_object_id='6cee0533-b9cd-462e-8a05-d781e8864f16' camera_name=1\n",
      "\tlaser_object_id='kWKrXcZMJ7I5K4Z4z9GdSQ' camera_object_id='7f37cfc3-1926-43a1-9af6-109ca0a5c797' camera_name=1\n",
      "\tlaser_object_id='sAuz7Nwbj2BM48D9VCq_zQ' camera_object_id='8142dfd0-b30b-4672-9833-3e48ddd7dec8' camera_name=1\n",
      "\tlaser_object_id='sJlmZW9yFdc8ca--Rhk7mw' camera_object_id='9df71c7e-d3ba-45dc-a107-60646d39dfb6' camera_name=1\n",
      "\tlaser_object_id='sJlmZW9yFdc8ca--Rhk7mw' camera_object_id='960d036e-c7ed-4e64-b069-86e114574d77' camera_name=3\n",
      "\tlaser_object_id='2-A6zakvKX2opVnyx9gplQ' camera_object_id='a6f937a6-7ea8-4393-b636-e0560e699856' camera_name=1\n",
      "\tlaser_object_id='2SYmRAjI0pCOwp2XYemMBQ' camera_object_id='ba670814-995e-4ade-bc42-58b0a1d8ec8d' camera_name=1\n",
      "\tlaser_object_id='2SYmRAjI0pCOwp2XYemMBQ' camera_object_id='ca9be338-79bb-4908-b4ee-5607a21b5b41' camera_name=2\n",
      "\tlaser_object_id='CB379cbEIKwW6eAs4wK6mw' camera_object_id='baae00a2-cacd-43c9-9d83-0932920c987e' camera_name=1\n",
      "\tlaser_object_id='l9f--u5kPKABASTzvy9Nug' camera_object_id='d4a7521f-1659-4ef3-a5ce-dc0811a223f6' camera_name=1\n",
      "\tlaser_object_id='oktALDz6ntRSXy5D7zFjEA' camera_object_id='d7539a85-ddfe-4b09-a7c7-13e5f702db6a' camera_name=1\n",
      "\tlaser_object_id='3mqY30kFisxCydiO2pKjPA' camera_object_id='e89cb911-600c-4b1a-a41a-b9681b76aa7d' camera_name=1\n",
      "\tlaser_object_id='OkBzhxzagapiYam_4NK-9A' camera_object_id='eff781d6-b34f-400b-95ad-b960fa0270d8' camera_name=1\n",
      "\tlaser_object_id='pATADVHYfotd4mWyfa9wVw' camera_object_id='7995b22b-fb0d-480d-89a1-c4fd0a37f852' camera_name=2\n",
      "\tlaser_object_id='NeHt8l-Xg_tze4fm4Zzhag' camera_object_id='fa0e471f-e5a4-4abc-a528-7be6ed3a1f9e' camera_name=2\n",
      "\tlaser_object_id='A4kzX4tBck4XcFuUOPA6Lg' camera_object_id='42d3fd29-cc31-44c6-a044-7af1a172488e' camera_name=3\n"
     ]
    }
   ],
   "source": [
    "#@title Sensor data with both lidar and camera boxes\n",
    "\n",
    "# Lazily read DataFrames for all components.\n",
    "association_df = read('camera_to_lidar_box_association')\n",
    "cam_box_df = read('camera_box')\n",
    "cam_img_df = read('camera_image')\n",
    "lidar_box_df = read('lidar_box')\n",
    "lidar_df = read('lidar')\n",
    "\n",
    "# Join all DataFrames using matching columns\n",
    "cam_image_w_box_df = v2.merge(cam_box_df, cam_img_df)\n",
    "cam_obj_df = v2.merge(association_df, cam_image_w_box_df)\n",
    "obj_df = v2.merge(cam_obj_df, lidar_box_df)\n",
    "# Group lidar sensors (left), group labels and camera images (right) and join.\n",
    "df = v2.merge(lidar_df, obj_df, left_group=True, right_group=True)\n",
    "\n",
    "# Read a single row, which contain data for all data for a single frame.\n",
    "_, row = next(iter(df.iterrows()))\n",
    "# Create all component objects\n",
    "camera_image = v2.CameraImageComponent.from_dict(row)\n",
    "lidar = v2.LiDARComponent.from_dict(row)\n",
    "camera_box = v2.CameraBoxComponent.from_dict(row)\n",
    "lidar_box = v2.LiDARBoxComponent.from_dict(row)\n",
    "\n",
    "print(\n",
    "    f'Found {len(lidar_box.key.laser_object_id)} objects on'\n",
    "    f' {lidar.key.segment_context_name=} {lidar.key.frame_timestamp_micros=}'\n",
    ")\n",
    "for laser_object_id, camera_object_id, camera_name in zip(\n",
    "    lidar_box.key.laser_object_id,\n",
    "    camera_box.key.camera_object_id,\n",
    "    camera_image.key.camera_name,\n",
    "):\n",
    "  print(f'\\t{laser_object_id=} {camera_object_id=} {camera_name=}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5gGBBIfiMX9-"
   },
   "source": [
    "In the example above the same camera images are repeated for multiple objects with the same camera_name. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "mNJQUlxRH7_Z",
    "outputId": "9e258ba2-b318-45d0-beca-c52e5fe860f1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 81 objects on lidar.key.segment_context_name='10023947602400723454_1120_000_1140_000' lidar.key.frame_timestamp_micros=1552440195362591\n",
      "\tlaser_object_id='-U88NMYnocLWCh6iqZwj1g' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='0VCoeT-jjrIfzTCsOWz20A' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='0_HBXNo3olLueqYvkPohlg' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='1nDCER_bA9py1ZPpNXecog' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='2-A6zakvKX2opVnyx9gplQ' camera_object_id='a6f937a6-7ea8-4393-b636-e0560e699856' camera_name=1.0\n",
      "\tlaser_object_id='2OYKagQRfCdaOXgU5RkMBA' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='2SYmRAjI0pCOwp2XYemMBQ' camera_object_id='ba670814-995e-4ade-bc42-58b0a1d8ec8d' camera_name=1.0\n",
      "\tlaser_object_id='2SYmRAjI0pCOwp2XYemMBQ' camera_object_id='ca9be338-79bb-4908-b4ee-5607a21b5b41' camera_name=2.0\n",
      "\tlaser_object_id='3083QteOhZ_vSpxmP0XK-Q' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='38Np8bwqcvw9KkrH3xHfpg' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='3I9vmrICjjnWtavq3ysJrQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='3mqY30kFisxCydiO2pKjPA' camera_object_id='e89cb911-600c-4b1a-a41a-b9681b76aa7d' camera_name=1.0\n",
      "\tlaser_object_id='45vGTk6HpmeCTR2Etiwq8w' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='63vjcxiQGxy6SmQ94yX-FQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='81faaqjwezt3wc0ZN-1N0g' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='8IZ7fkXm0FDeUdEuJAqSlA' camera_object_id='260f57c0-787d-4f36-bfe9-64ca9828448f' camera_name=1.0\n",
      "\tlaser_object_id='8k1Wkk08druqr8A5YTrqFQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='A4kzX4tBck4XcFuUOPA6Lg' camera_object_id='42d3fd29-cc31-44c6-a044-7af1a172488e' camera_name=3.0\n",
      "\tlaser_object_id='AWjsPzPKZ7MTv8Ta3AXBbQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='AhVLDKKFO3nP9YBAG8x0pw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='CB379cbEIKwW6eAs4wK6mw' camera_object_id='baae00a2-cacd-43c9-9d83-0932920c987e' camera_name=1.0\n",
      "\tlaser_object_id='DMyvJcDBgX_UfZY7i4gQUg' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='EKH1nF9LYmAF7RP2zcpTAw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='FF2DYhbJHb6U66o94jP48Q' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='FaLjVnzgLbkn0NFfVlonZw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='GZswbdmDQZSe88ajnioK_g' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='Iv3M96stRJ3JE-umZR0uVA' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='JyfDMV88pTVkjurlKpSYsw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='Kd-ybN2790ngkQSEC0NNkQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='L5HBM5tSKtNA4qHegDQk8Q' camera_object_id='4b9a9206-fbf1-4d45-8765-197759100bd6' camera_name=1.0\n",
      "\tlaser_object_id='LubISPcZqG5jjc-d4fOXBQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='MGw5mrKnm4f98o1li921HQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='NeHt8l-Xg_tze4fm4Zzhag' camera_object_id='fa0e471f-e5a4-4abc-a528-7be6ed3a1f9e' camera_name=2.0\n",
      "\tlaser_object_id='Nx3jDnV5BFvG2rNbAk93sw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='OTxlocnBbws-LpK8i2X10w' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='OkBzhxzagapiYam_4NK-9A' camera_object_id='eff781d6-b34f-400b-95ad-b960fa0270d8' camera_name=1.0\n",
      "\tlaser_object_id='PrS_b18HNPEbxJ9IAYW2BQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='SdPkKCcb4GwSs_dXFeDfKA' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='SkvDD99-XNR9FxtiMhbpww' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='W5rhCNFFOR-wpcMvRNEM1A' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='WrQ9WB5nz7yw3pLq2Sx4JA' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='YTw2k0_-8UqolJn4WQRc2g' camera_object_id='699b6cb3-afc1-469f-8ac9-40978aeb511b' camera_name=1.0\n",
      "\tlaser_object_id='YZp0XjCF9MO6a2x7szWEXA' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='Yyu039jUMIJ4gI_2-mTkSg' camera_object_id='6cee0533-b9cd-462e-8a05-d781e8864f16' camera_name=1.0\n",
      "\tlaser_object_id='ZyK_iICxQsEYdJLxMFTw7w' camera_object_id='08b1e19e-b912-4963-a7bb-55e138bbc25e' camera_name=1.0\n",
      "\tlaser_object_id='ZyK_iICxQsEYdJLxMFTw7w' camera_object_id='da518036-17cb-48c4-97aa-97939f989d16' camera_name=3.0\n",
      "\tlaser_object_id='aZNY65GX0cPUkxaWoiYiFA' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='brLOJzgVMhP_-kdQzIa7ng' camera_object_id='30326915-a157-4c60-9bef-0a188938c998' camera_name=1.0\n",
      "\tlaser_object_id='dNW3S4yA8s8GmltmAPV8LQ' camera_object_id='24508c28-3154-4b7f-964d-70b3ed7d5a9d' camera_name=1.0\n",
      "\tlaser_object_id='dWu2dkV959ZVbiXuXYme9A' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='eTLt3z7Tt1__uHa0Pj96uQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='g_YHfVchyJsQkIwc7pzcvw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='hVOvDRfPPYheooNcglGZfw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='hhuZm4IrFKySygbA_3k80w' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='iYoNqzfcHozgqJVHl7Yxnw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='jDO7AFvKzSj9IBX0gjUdVw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='jYdxjpZOYqJdSiBQI133Fw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='k7sk2VXkecnWsXdCfg3Wig' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='kWKrXcZMJ7I5K4Z4z9GdSQ' camera_object_id='7f37cfc3-1926-43a1-9af6-109ca0a5c797' camera_name=1.0\n",
      "\tlaser_object_id='l9f--u5kPKABASTzvy9Nug' camera_object_id='d4a7521f-1659-4ef3-a5ce-dc0811a223f6' camera_name=1.0\n",
      "\tlaser_object_id='mOkkhCfy9ezsL1Xpiin0cg' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='oktALDz6ntRSXy5D7zFjEA' camera_object_id='d7539a85-ddfe-4b09-a7c7-13e5f702db6a' camera_name=1.0\n",
      "\tlaser_object_id='oq32CII6BSHT6k4uzf5XrA' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='pATADVHYfotd4mWyfa9wVw' camera_object_id='7995b22b-fb0d-480d-89a1-c4fd0a37f852' camera_name=2.0\n",
      "\tlaser_object_id='pJ3fmc4NHVFkhDPgGBtS4Q' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='qlky7ZEPOYxqaAjh3R6lPQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='r1imeuTUWgH_ZJXDYGvC8g' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='rM0XsOkipsd4-TUUQ5ncsQ' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='rrONOa8AiwnDsrF8mD8-fw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='sAuz7Nwbj2BM48D9VCq_zQ' camera_object_id='8142dfd0-b30b-4672-9833-3e48ddd7dec8' camera_name=1.0\n",
      "\tlaser_object_id='sJlmZW9yFdc8ca--Rhk7mw' camera_object_id='9df71c7e-d3ba-45dc-a107-60646d39dfb6' camera_name=1.0\n",
      "\tlaser_object_id='sJlmZW9yFdc8ca--Rhk7mw' camera_object_id='960d036e-c7ed-4e64-b069-86e114574d77' camera_name=3.0\n",
      "\tlaser_object_id='t9E-elYgX_LpXadnao0H3Q' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='tmWtGvgp0B6MjAE2qZuD9w' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='txvVqORJ6Gcf510DdJvW4Q' camera_object_id='4286a233-4a34-4d99-b214-2d8690e92570' camera_name=1.0\n",
      "\tlaser_object_id='u13bFbFrNUgtlke3hkv95Q' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='uTQJwYsr8nw8M5kF2ELaiA' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='wO7bCMlAPPxIZSpihCgJBw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='wcj6Uo3otoyeYSL_80w-Rw' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='x3d22mf0934NCWzNdz1CVg' camera_object_id=nan camera_name=nan\n",
      "\tlaser_object_id='zW21zQ-BE0degWh8LKLVXw' camera_object_id=nan camera_name=nan\n"
     ]
    }
   ],
   "source": [
    "# @title Sensor data with lidar and camera boxes if available\n",
    "\n",
    "# Lazily read DataFrames for all components.\n",
    "association_df = read('camera_to_lidar_box_association')\n",
    "cam_box_df = read('camera_box')\n",
    "cam_img_df = read('camera_image')\n",
    "lidar_box_df = read('lidar_box')\n",
    "lidar_df = read('lidar')\n",
    "\n",
    "# Join all DataFrames using matching columns\n",
    "cam_image_w_box_df = v2.merge(cam_box_df, cam_img_df)\n",
    "cam_obj_df = v2.merge(association_df, cam_image_w_box_df)\n",
    "# In this example camera box labels are optional, so we set left_nullable=True.\n",
    "obj_df = v2.merge(cam_obj_df, lidar_box_df, left_nullable=True)\n",
    "# Group lidar sensors (left), group labels and camera images (right) and join.\n",
    "df = v2.merge(lidar_df, obj_df, left_group=True, right_group=True)\n",
    "\n",
    "# Read a single row, which contain data for all data for a single frame.\n",
    "_, row = next(iter(df.iterrows()))\n",
    "# Create all component objects\n",
    "camera_image = v2.CameraImageComponent.from_dict(row)\n",
    "lidar = v2.LiDARComponent.from_dict(row)\n",
    "camera_box = v2.CameraBoxComponent.from_dict(row)\n",
    "lidar_box = v2.LiDARBoxComponent.from_dict(row)\n",
    "\n",
    "print(\n",
    "    f'Found {len(lidar_box.key.laser_object_id)} objects on'\n",
    "    f' {lidar.key.segment_context_name=} {lidar.key.frame_timestamp_micros=}'\n",
    ")\n",
    "for laser_object_id, camera_object_id, camera_name in zip(\n",
    "    lidar_box.key.laser_object_id,\n",
    "    camera_box.key.camera_object_id,\n",
    "    camera_image.key.camera_name,\n",
    "):\n",
    "  print(f'\\t{laser_object_id=} {camera_object_id=} {camera_name=}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "56O0MMq-DQ2E"
   },
   "source": [
    "### Object level\n",
    "\n",
    "At this level each row of a table represents a single object at a single frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "Ucv2uKXmHr_K"
   },
   "outputs": [],
   "source": [
    "# @title Camera image, lidar, boxes with keypoints\n",
    "\n",
    "\n",
    "# Lazily read DataFrames for all components.\n",
    "association_df = read('camera_to_lidar_box_association')\n",
    "cam_hkp_df = read('camera_hkp')\n",
    "cam_box_df = read('camera_box')\n",
    "cam_img_df = read('camera_image')\n",
    "lidar_box_df = read('lidar_box')\n",
    "lidar_df = read('lidar')\n",
    "\n",
    "# Join all DataFrame objects for all components together.\n",
    "cam_image_w_box_df = v2.merge(cam_box_df, cam_img_df)\n",
    "cam_image_w_box_w_hkp_df = v2.merge(cam_image_w_box_df, cam_hkp_df)\n",
    "cam_obj_df = v2.merge(association_df, cam_image_w_box_w_hkp_df)\n",
    "obj_df = v2.merge(cam_obj_df, lidar_box_df)\n",
    "t = v2.merge(lidar_df, obj_df, left_group=True, right_group=True)\n",
    "\n",
    "# Create a row iterator (continue to the next cell)\n",
    "it = iter(t.iterrows())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "Gg8Ro6BZqxqR",
    "outputId": "38dbf57a-e277-4be2-afd8-9c01dcb3de17"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 17 objects on lidar.key.segment_context_name='10023947602400723454_1120_000_1140_000' lidar.key.frame_timestamp_micros=1552440196462383\n",
      "\tlaser_object_id='dNW3S4yA8s8GmltmAPV8LQ' camera_object_id='24508c28-3154-4b7f-964d-70b3ed7d5a9d' camera_name=1 with 6 camera keypoints\n",
      "\tlaser_object_id='8IZ7fkXm0FDeUdEuJAqSlA' camera_object_id='260f57c0-787d-4f36-bfe9-64ca9828448f' camera_name=1 with 14 camera keypoints\n",
      "\tlaser_object_id='brLOJzgVMhP_-kdQzIa7ng' camera_object_id='30326915-a157-4c60-9bef-0a188938c998' camera_name=1 with 11 camera keypoints\n",
      "\tlaser_object_id='YTw2k0_-8UqolJn4WQRc2g' camera_object_id='699b6cb3-afc1-469f-8ac9-40978aeb511b' camera_name=1 with 11 camera keypoints\n",
      "\tlaser_object_id='Yyu039jUMIJ4gI_2-mTkSg' camera_object_id='6cee0533-b9cd-462e-8a05-d781e8864f16' camera_name=1 with 10 camera keypoints\n",
      "\tlaser_object_id='kWKrXcZMJ7I5K4Z4z9GdSQ' camera_object_id='7f37cfc3-1926-43a1-9af6-109ca0a5c797' camera_name=1 with 6 camera keypoints\n",
      "\tlaser_object_id='sJlmZW9yFdc8ca--Rhk7mw' camera_object_id='9df71c7e-d3ba-45dc-a107-60646d39dfb6' camera_name=1 with 14 camera keypoints\n",
      "\tlaser_object_id='sJlmZW9yFdc8ca--Rhk7mw' camera_object_id='960d036e-c7ed-4e64-b069-86e114574d77' camera_name=3 with 14 camera keypoints\n",
      "\tlaser_object_id='2-A6zakvKX2opVnyx9gplQ' camera_object_id='a6f937a6-7ea8-4393-b636-e0560e699856' camera_name=1 with 12 camera keypoints\n",
      "\tlaser_object_id='2SYmRAjI0pCOwp2XYemMBQ' camera_object_id='ba670814-995e-4ade-bc42-58b0a1d8ec8d' camera_name=1 with 2 camera keypoints\n",
      "\tlaser_object_id='l9f--u5kPKABASTzvy9Nug' camera_object_id='d4a7521f-1659-4ef3-a5ce-dc0811a223f6' camera_name=1 with 14 camera keypoints\n",
      "\tlaser_object_id='oktALDz6ntRSXy5D7zFjEA' camera_object_id='d7539a85-ddfe-4b09-a7c7-13e5f702db6a' camera_name=1 with 8 camera keypoints\n",
      "\tlaser_object_id='3mqY30kFisxCydiO2pKjPA' camera_object_id='e89cb911-600c-4b1a-a41a-b9681b76aa7d' camera_name=1 with 11 camera keypoints\n",
      "\tlaser_object_id='OkBzhxzagapiYam_4NK-9A' camera_object_id='eff781d6-b34f-400b-95ad-b960fa0270d8' camera_name=1 with 14 camera keypoints\n",
      "\tlaser_object_id='OkBzhxzagapiYam_4NK-9A' camera_object_id='08ac1a40-1f4d-42a5-b3a5-24ea616d58dc' camera_name=3 with 14 camera keypoints\n",
      "\tlaser_object_id='A4kzX4tBck4XcFuUOPA6Lg' camera_object_id='42d3fd29-cc31-44c6-a044-7af1a172488e' camera_name=3 with 6 camera keypoints\n",
      "\tlaser_object_id='ZyK_iICxQsEYdJLxMFTw7w' camera_object_id='da518036-17cb-48c4-97aa-97939f989d16' camera_name=3 with 11 camera keypoints\n"
     ]
    }
   ],
   "source": [
    "# Execute this cell multiple times to see data for different rows.\n",
    "\n",
    "# Actually read the data.\n",
    "_, row = next(it)\n",
    "\n",
    "# Create all component objects\n",
    "camera_hkp = v2.CameraHumanKeypointsComponent.from_dict(row)\n",
    "camera_box = v2.CameraBoxComponent.from_dict(row)\n",
    "camera_image = v2.CameraImageComponent.from_dict(row)\n",
    "lidar_box = v2.LiDARBoxComponent.from_dict(row)\n",
    "lidar = v2.LiDARComponent.from_dict(row)\n",
    "\n",
    "print(\n",
    "    f'Found {len(lidar_box.key.laser_object_id)} objects on'\n",
    "    f' {lidar.key.segment_context_name=} {lidar.key.frame_timestamp_micros=}'\n",
    ")\n",
    "for laser_object_id, camera_object_id, camera_name, cam_kp_x in zip(\n",
    "    lidar_box.key.laser_object_id,\n",
    "    camera_box.key.camera_object_id,\n",
    "    camera_image.key.camera_name,\n",
    "    camera_hkp.camera_keypoints.keypoint_2d.location_px.x,\n",
    "):\n",
    "  print(\n",
    "      f'\\t{laser_object_id=} {camera_object_id=} {camera_name=} with'\n",
    "      f' {len(cam_kp_x)} camera keypoints'\n",
    "  )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8PT8IXmMzBt7"
   },
   "source": [
    "# Dataset Format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IWR8_hets6c7"
   },
   "source": [
    "## Folder structure\n",
    "\n",
    "Here are a few examples of how the file paths might look using the format `{path_to_the_dataset}/{component_tag}/{component_name}.parquet`:\n",
    "\n",
    "```\n",
    "/waymo_open_dataset/camera_image/10023947602400723454_1120_000_1140_000.parquet\n",
    "...\n",
    "/waymo_open_dataset/lidar_box/10023947602400723454_1120_000_1140_000.parquet\n",
    "...\n",
    "/waymo_open_dataset/lidar/10023947602400723454_1120_000_1140_000.parquet\n",
    "```\n",
    "\n",
    "In these examples, \"waymo_open_dataset\" is the path to the dataset, \"camera_image\", \"lidar_box\", and \"lidar\" are the string tags for corresponding components, \"10023947602400723454_1120_000_1140_000\" is the `context_name`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fG3wayza3RFy"
   },
   "source": [
    "## Loading raw data\n",
    "\n",
    "You can use any existing library which supports the Apache Parquet files to read the dataset, for example PyArrow, Pandas, Dask or any other. We recommend to use [Dask](https://docs.dask.org/en/stable/) to access entire dataset, because it supports larger-then-memory tables and distributed processing. If data for a single segment (aka partition) fit into memory, you can use Pandas as well. Both libraries have very similar APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "height": 374
    },
    "id": "nB5mRt6i4fIS",
    "outputId": "a74bffdb-5e07-4cd2-84b4-42ce06abc434"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading a single shard from a single component /tmp/wod_debug_20230212/camera_image/10023947602400723454_1120_000_1140_000.parquet\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-042a892f-7a0f-4928-be97-149c39572c34\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key.segment_context_name</th>\n",
       "      <th>key.frame_timestamp_micros</th>\n",
       "      <th>key.camera_name</th>\n",
       "      <th>[CameraImageComponent].image</th>\n",
       "      <th>[CameraImageComponent].pose.transform</th>\n",
       "      <th>[CameraImageComponent].velocity.linear_velocity.x</th>\n",
       "      <th>[CameraImageComponent].velocity.linear_velocity.y</th>\n",
       "      <th>[CameraImageComponent].velocity.linear_velocity.z</th>\n",
       "      <th>[CameraImageComponent].velocity.angular_velocity.x</th>\n",
       "      <th>[CameraImageComponent].velocity.angular_velocity.y</th>\n",
       "      <th>[CameraImageComponent].velocity.angular_velocity.z</th>\n",
       "      <th>[CameraImageComponent].pose_timestamp</th>\n",
       "      <th>[CameraImageComponent].rolling_shutter_params.shutter</th>\n",
       "      <th>[CameraImageComponent].rolling_shutter_params.camera_trigger_time</th>\n",
       "      <th>[CameraImageComponent].rolling_shutter_params.camera_readout_done_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10023947602400723454_1120_000_1140_000</td>\n",
       "      <td>1552440195362591</td>\n",
       "      <td>1</td>\n",
       "      <td>b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...</td>\n",
       "      <td>[0.6990588408049747, 0.7150334085392328, 0.006...</td>\n",
       "      <td>1.337475</td>\n",
       "      <td>-1.359002</td>\n",
       "      <td>0.027288</td>\n",
       "      <td>-0.015392</td>\n",
       "      <td>0.012218</td>\n",
       "      <td>0.011145</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.006981</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10023947602400723454_1120_000_1140_000</td>\n",
       "      <td>1552440195362591</td>\n",
       "      <td>2</td>\n",
       "      <td>b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...</td>\n",
       "      <td>[0.6989900519876013, 0.7151025083199268, 0.006...</td>\n",
       "      <td>1.333927</td>\n",
       "      <td>-1.373261</td>\n",
       "      <td>0.024719</td>\n",
       "      <td>-0.016407</td>\n",
       "      <td>0.010103</td>\n",
       "      <td>0.017257</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.009992</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10023947602400723454_1120_000_1140_000</td>\n",
       "      <td>1552440195362591</td>\n",
       "      <td>4</td>\n",
       "      <td>b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...</td>\n",
       "      <td>[0.6987839219647801, 0.7153068814214333, 0.006...</td>\n",
       "      <td>1.339028</td>\n",
       "      <td>-1.374271</td>\n",
       "      <td>0.019406</td>\n",
       "      <td>-0.017616</td>\n",
       "      <td>0.006203</td>\n",
       "      <td>0.013990</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.009992</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.210402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10023947602400723454_1120_000_1140_000</td>\n",
       "      <td>1552440195362591</td>\n",
       "      <td>3</td>\n",
       "      <td>b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...</td>\n",
       "      <td>[0.6991963373797294, 0.7148964117436919, 0.006...</td>\n",
       "      <td>1.332058</td>\n",
       "      <td>-1.358936</td>\n",
       "      <td>0.031292</td>\n",
       "      <td>-0.015262</td>\n",
       "      <td>0.015056</td>\n",
       "      <td>0.013473</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.009992</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.071130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10023947602400723454_1120_000_1140_000</td>\n",
       "      <td>1552440195362591</td>\n",
       "      <td>5</td>\n",
       "      <td>b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...</td>\n",
       "      <td>[0.6992650451400215, 0.7148278952390329, 0.007...</td>\n",
       "      <td>1.332926</td>\n",
       "      <td>-1.353070</td>\n",
       "      <td>0.035072</td>\n",
       "      <td>-0.014256</td>\n",
       "      <td>0.017961</td>\n",
       "      <td>0.010754</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.009992</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.537538</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-042a892f-7a0f-4928-be97-149c39572c34')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-042a892f-7a0f-4928-be97-149c39572c34 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-042a892f-7a0f-4928-be97-149c39572c34');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                 key.segment_context_name  key.frame_timestamp_micros  key.camera_name                       [CameraImageComponent].image  ... [CameraImageComponent].rolling_shutter_params.shutter  [CameraImageComponent].rolling_shutter_params.camera_trigger_time  [CameraImageComponent].rolling_shutter_params.camera_readout_done_time\n",
       "0  10023947602400723454_1120_000_1140_000            1552440195362591                1  b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...  ...                                           0.006981                                          1.552440e+09                                                       1.552440e+09\n",
       "1  10023947602400723454_1120_000_1140_000            1552440195362591                2  b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...  ...                                           0.009992                                          1.552440e+09                                                       1.552440e+09\n",
       "2  10023947602400723454_1120_000_1140_000            1552440195362591                4  b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...  ...                                           0.009992                                          1.552440e+09                                                       1.552440e+09\n",
       "3  10023947602400723454_1120_000_1140_000            1552440195362591                3  b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...  ...                                           0.009992                                          1.552440e+09                                                       1.552440e+09\n",
       "4  10023947602400723454_1120_000_1140_000            1552440195362591                5  b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...  ...                                           0.009992                                          1.552440e+09                                                       1.552440e+09\n",
       "\n",
       "[5 rows x 16 columns]"
      ]
     },
     "execution_count": 0,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import dask.dataframe as dd\n",
    "\n",
    "table_path = f'{dataset_dir}/camera_image/{context_name}.parquet'\n",
    "print(f'Reading a single shard from a single component {table_path}')\n",
    "table = dd.read_parquet(table_path)\n",
    "table.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8xI2LSqbwebI"
   },
   "source": [
    "## Supported components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mPiTE2NCwtaY"
   },
   "source": [
    "A **component** in the context of the Waymo Open Dataset v2 format is a set of related fields (aka table columns) that are all needed to understand each individual field. In other words, a component is a collection of data that is related and interdependent. For example, camera image component includes the following fields:\n",
    "\n",
    " type | Column name\n",
    " --- | ---\n",
    " string | key.segment_context_name\n",
    " int64 | key.frame_timestamp_micros\n",
    " int8 | key.camera_name\n",
    " binary | [CameraImageComponent].image\n",
    " fixed_size_list<item: double>[16] | [CameraImageComponent].pose.transform\n",
    " float | [CameraImageComponent].velocity.linear_velocity.x\n",
    " float | [CameraImageComponent].velocity.linear_velocity.y\n",
    " float | [CameraImageComponent].velocity.linear_velocity.z\n",
    " double | [CameraImageComponent].velocity.angular_velocity.x\n",
    " double | [CameraImageComponent].velocity.angular_velocity.y\n",
    " double | [CameraImageComponent].velocity.angular_velocity.z\n",
    " double | [CameraImageComponent].pose_timestamp\n",
    " double | [CameraImageComponent].rolling_shutter_params.shutter\n",
    " double | [CameraImageComponent].rolling_shutter_params.camera_trigger_time\n",
    " double | [CameraImageComponent].rolling_shutter_params.camera_readout_done_time\n",
    "\n",
    "\n",
    "Key columns start with `key.` prefix and use same names across all components. Component specific columns start with `[Name of a Component].` prefix. Each column represents a simple Python type or a list of elements of a simple type.\n",
    "For example a 3D vector representing linear velocity of a camera sensor at the\n",
    "specific frame is stored in 3 table columns:\n",
    "\n",
    "\n",
    " type | Column name\n",
    " --- | ---\n",
    "float | [CameraImageComponent].velocity.linear_velocity.x\n",
    "float | [CameraImageComponent].velocity.linear_velocity.y\n",
    "float | [CameraImageComponent].velocity.linear_velocity.z\n",
    "\n",
    "For the details about each component refer to the source code under `waymo_open_dataset/v2/perception`. To see all available components execute the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "id": "vY_whUMQIdha",
    "outputId": "d0e89319-4943-4703-b83b-8109bb969fd6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available components:\n",
      "camera_box: CameraBoxComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tint8                                     key.camera_name\n",
      "\tstring                                   key.camera_object_id\n",
      "\tdouble                                   [CameraBoxComponent].box.center.x\n",
      "\tdouble                                   [CameraBoxComponent].box.center.y\n",
      "\tdouble                                   [CameraBoxComponent].box.size.x\n",
      "\tdouble                                   [CameraBoxComponent].box.size.y\n",
      "\tint8                                     [CameraBoxComponent].type\n",
      "\tint8                                     [CameraBoxComponent].difficulty_level.detection\n",
      "\tint8                                     [CameraBoxComponent].difficulty_level.tracking\n",
      "camera_calibration: CameraCalibrationComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint8                                     key.camera_name\n",
      "\tdouble                                   [CameraCalibrationComponent].intrinsic.f_u\n",
      "\tdouble                                   [CameraCalibrationComponent].intrinsic.f_v\n",
      "\tdouble                                   [CameraCalibrationComponent].intrinsic.c_u\n",
      "\tdouble                                   [CameraCalibrationComponent].intrinsic.c_v\n",
      "\tdouble                                   [CameraCalibrationComponent].intrinsic.k1\n",
      "\tdouble                                   [CameraCalibrationComponent].intrinsic.k2\n",
      "\tdouble                                   [CameraCalibrationComponent].intrinsic.p1\n",
      "\tdouble                                   [CameraCalibrationComponent].intrinsic.p2\n",
      "\tdouble                                   [CameraCalibrationComponent].intrinsic.k3\n",
      "\tfixed_size_list<item: double>[16]        [CameraCalibrationComponent].extrinsic.transform\n",
      "\tint32                                    [CameraCalibrationComponent].width\n",
      "\tint32                                    [CameraCalibrationComponent].height\n",
      "\tint8                                     [CameraCalibrationComponent].rolling_shutter_direction\n",
      "camera_hkp: CameraHumanKeypointsComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tint8                                     key.camera_name\n",
      "\tstring                                   key.camera_object_id\n",
      "\tlist<item: int8>                         [CameraHumanKeypointsComponent].camera_keypoints[*].type\n",
      "\tlist<item: double>                       [CameraHumanKeypointsComponent].camera_keypoints[*].keypoint_2d.location_px.x\n",
      "\tlist<item: double>                       [CameraHumanKeypointsComponent].camera_keypoints[*].keypoint_2d.location_px.y\n",
      "\tlist<item: bool>                         [CameraHumanKeypointsComponent].camera_keypoints[*].keypoint_2d.visibility.is_occluded\n",
      "\tlist<item: double>                       [CameraHumanKeypointsComponent].camera_keypoints[*].keypoint_3d.location_m.x\n",
      "\tlist<item: double>                       [CameraHumanKeypointsComponent].camera_keypoints[*].keypoint_3d.location_m.y\n",
      "\tlist<item: double>                       [CameraHumanKeypointsComponent].camera_keypoints[*].keypoint_3d.location_m.z\n",
      "\tlist<item: bool>                         [CameraHumanKeypointsComponent].camera_keypoints[*].keypoint_3d.visibility.is_occluded\n",
      "camera_image: CameraImageComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tint8                                     key.camera_name\n",
      "\tbinary                                   [CameraImageComponent].image\n",
      "\tfixed_size_list<item: double>[16]        [CameraImageComponent].pose.transform\n",
      "\tfloat                                    [CameraImageComponent].velocity.linear_velocity.x\n",
      "\tfloat                                    [CameraImageComponent].velocity.linear_velocity.y\n",
      "\tfloat                                    [CameraImageComponent].velocity.linear_velocity.z\n",
      "\tdouble                                   [CameraImageComponent].velocity.angular_velocity.x\n",
      "\tdouble                                   [CameraImageComponent].velocity.angular_velocity.y\n",
      "\tdouble                                   [CameraImageComponent].velocity.angular_velocity.z\n",
      "\tdouble                                   [CameraImageComponent].pose_timestamp\n",
      "\tdouble                                   [CameraImageComponent].rolling_shutter_params.shutter\n",
      "\tdouble                                   [CameraImageComponent].rolling_shutter_params.camera_trigger_time\n",
      "\tdouble                                   [CameraImageComponent].rolling_shutter_params.camera_readout_done_time\n",
      "camera_to_lidar_box_association: CameraToLiDARBoxAssociationComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tint8                                     key.camera_name\n",
      "\tstring                                   key.camera_object_id\n",
      "\tstring                                   key.laser_object_id\n",
      "lidar_box: LiDARBoxComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tstring                                   key.laser_object_id\n",
      "\tdouble                                   [LiDARBoxComponent].box.center.x\n",
      "\tdouble                                   [LiDARBoxComponent].box.center.y\n",
      "\tdouble                                   [LiDARBoxComponent].box.center.z\n",
      "\tdouble                                   [LiDARBoxComponent].box.size.x\n",
      "\tdouble                                   [LiDARBoxComponent].box.size.y\n",
      "\tdouble                                   [LiDARBoxComponent].box.size.z\n",
      "\tdouble                                   [LiDARBoxComponent].box.heading\n",
      "\tint8                                     [LiDARBoxComponent].type\n",
      "\tdouble                                   [LiDARBoxComponent].speed.x\n",
      "\tdouble                                   [LiDARBoxComponent].speed.y\n",
      "\tdouble                                   [LiDARBoxComponent].speed.z\n",
      "\tdouble                                   [LiDARBoxComponent].acceleration.x\n",
      "\tdouble                                   [LiDARBoxComponent].acceleration.y\n",
      "\tdouble                                   [LiDARBoxComponent].acceleration.z\n",
      "\tint64                                    [LiDARBoxComponent].num_lidar_points_in_box\n",
      "\tint64                                    [LiDARBoxComponent].num_top_lidar_points_in_box\n",
      "\tint8                                     [LiDARBoxComponent].difficulty_level.detection\n",
      "\tint8                                     [LiDARBoxComponent].difficulty_level.tracking\n",
      "lidar_calibration: LiDARCalibrationComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint8                                     key.laser_name\n",
      "\tfixed_size_list<item: double>[16]        [LiDARCalibrationComponent].extrinsic.transform\n",
      "\tdouble                                   [LiDARCalibrationComponent].beam_inclination.min\n",
      "\tdouble                                   [LiDARCalibrationComponent].beam_inclination.max\n",
      "\tlist<item: double>                       [LiDARCalibrationComponent].beam_inclination.values\n",
      "lidar_camera_projection: LiDARCameraProjectionComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tint8                                     key.laser_name\n",
      "\tlist<item: float>                        [LiDARCameraProjectionComponent].range_image_return1.values\n",
      "\tfixed_size_list<item: int32>[3]          [LiDARCameraProjectionComponent].range_image_return1.shape\n",
      "\tlist<item: float>                        [LiDARCameraProjectionComponent].range_image_return2.values\n",
      "\tfixed_size_list<item: int32>[3]          [LiDARCameraProjectionComponent].range_image_return2.shape\n",
      "lidar_camera_synced_box: LiDARCameraSyncedBoxComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tstring                                   key.laser_object_id\n",
      "\tint8                                     [LiDARCameraSyncedBoxComponent].most_visible_camera_name\n",
      "\tdouble                                   [LiDARCameraSyncedBoxComponent].camera_synced_box.center.x\n",
      "\tdouble                                   [LiDARCameraSyncedBoxComponent].camera_synced_box.center.y\n",
      "\tdouble                                   [LiDARCameraSyncedBoxComponent].camera_synced_box.center.z\n",
      "\tdouble                                   [LiDARCameraSyncedBoxComponent].camera_synced_box.size.x\n",
      "\tdouble                                   [LiDARCameraSyncedBoxComponent].camera_synced_box.size.y\n",
      "\tdouble                                   [LiDARCameraSyncedBoxComponent].camera_synced_box.size.z\n",
      "\tdouble                                   [LiDARCameraSyncedBoxComponent].camera_synced_box.heading\n",
      "lidar: LiDARComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tint8                                     key.laser_name\n",
      "\tlist<item: float>                        [LiDARComponent].range_image_return1.values\n",
      "\tfixed_size_list<item: int32>[3]          [LiDARComponent].range_image_return1.shape\n",
      "\tlist<item: float>                        [LiDARComponent].range_image_return2.values\n",
      "\tfixed_size_list<item: int32>[3]          [LiDARComponent].range_image_return2.shape\n",
      "lidar_hkp: LiDARHumanKeypointsComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tstring                                   key.laser_object_id\n",
      "\tlist<item: int8>                         [LiDARHumanKeypointsComponent].lidar_keypoints[*].type\n",
      "\tlist<item: double>                       [LiDARHumanKeypointsComponent].lidar_keypoints[*].keypoint_3d.location_m.x\n",
      "\tlist<item: double>                       [LiDARHumanKeypointsComponent].lidar_keypoints[*].keypoint_3d.location_m.y\n",
      "\tlist<item: double>                       [LiDARHumanKeypointsComponent].lidar_keypoints[*].keypoint_3d.location_m.z\n",
      "\tlist<item: bool>                         [LiDARHumanKeypointsComponent].lidar_keypoints[*].keypoint_3d.visibility.is_occluded\n",
      "lidar_pose: LiDARPoseComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tint8                                     key.laser_name\n",
      "\tlist<item: float>                        [LiDARPoseComponent].range_image_return1.values\n",
      "\tfixed_size_list<item: int32>[3]          [LiDARPoseComponent].range_image_return1.shape\n",
      "projected_lidar_box: ProjectedLiDARBoxComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tint8                                     key.camera_name\n",
      "\tstring                                   key.laser_object_id\n",
      "\tdouble                                   [ProjectedLiDARBoxComponent].box.center.x\n",
      "\tdouble                                   [ProjectedLiDARBoxComponent].box.center.y\n",
      "\tdouble                                   [ProjectedLiDARBoxComponent].box.size.x\n",
      "\tdouble                                   [ProjectedLiDARBoxComponent].box.size.y\n",
      "\tint8                                     [ProjectedLiDARBoxComponent].type\n",
      "stats: StatsComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tstring                                   [StatsComponent].time_of_day\n",
      "\tstring                                   [StatsComponent].location\n",
      "\tstring                                   [StatsComponent].weather\n",
      "\tlist<item: int8>                         [StatsComponent].lidar_object_counts.types\n",
      "\tlist<item: int32>                        [StatsComponent].lidar_object_counts.counts\n",
      "\tlist<item: int8>                         [StatsComponent].camera_object_counts.types\n",
      "\tlist<item: int32>                        [StatsComponent].camera_object_counts.counts\n",
      "vehicle_pose: VehiclePoseComponent\n",
      "\tstring                                   key.segment_context_name\n",
      "\tint64                                    key.frame_timestamp_micros\n",
      "\tfixed_size_list<item: double>[16]        [VehiclePoseComponent].world_from_vehicle.transform\n"
     ]
    }
   ],
   "source": [
    "# @title List columns in all components\n",
    "\n",
    "print('Available components:')\n",
    "for component, tag in v2.TAG_BY_COMPONENT.items():\n",
    "  print(f'{tag}: {component.__name__}')\n",
    "  schema = component.schema()\n",
    "  for column, arrow_type in zip(schema.names, schema.types):\n",
    "    print(f'\\t{str(arrow_type):40s} {column}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X2Acmm7j6dQx"
   },
   "source": [
    "## Object-oriented API\n",
    "\n",
    "Users can access the dataset using the mentioned external libraries, but the WODv2 python library defines an object-oriented API - a high level abstraction that allows users to work with the data in a more intuitive and streamlined manner. The API provides easy-to-use classes and methods for accessing and manipulating the data, making it easier to write code and reducing the time required to perform complex data operations. \n",
    "\n",
    "Below are two examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "j2cRE4M36bnl",
    "outputId": "48b9054f-9c0c-454f-cde2-faab2dc1be55"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x=1.3374745845794678 y=-1.359001636505127 z=0.02728821150958538\n",
      "p.x=1.3374745845794678 p.y=-1.359001636505127 p.z=0.02728821150958538\n"
     ]
    }
   ],
   "source": [
    "# Read a single row from the table\n",
    "_, row = next(iter(table.iterrows()))\n",
    "\n",
    "\n",
    "# Direct access to the tabular data\n",
    "def process_coordinates(x: float, y: float, z: float) -> None:\n",
    "  print(f'{x=} {y=} {z=}')\n",
    "\n",
    "# Use DataFrame row directly\n",
    "process_coordinates(\n",
    "    row['[CameraImageComponent].velocity.linear_velocity.x'],\n",
    "    row['[CameraImageComponent].velocity.linear_velocity.y'],\n",
    "    row['[CameraImageComponent].velocity.linear_velocity.z'],\n",
    ")\n",
    "\n",
    "\n",
    "# Access via the object-oriented API\n",
    "def process_point(p: v2.Vec3d) -> None:\n",
    "  print(f'{p.x=} {p.y=} {p.z=}')\n",
    "\n",
    "image = v2.CameraImageComponent.from_dict(row)\n",
    "process_point(image.velocity.linear_velocity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u571xp_TzWXK"
   },
   "source": [
    "## A relational database-like structure\n",
    "\n",
    "The dataset is organized into multiple tables with multi-column keys, creating a relational database-like structure on top of Apache Parquet files. This structure combines the benefits of Apache Parquet's efficient storage and retrieval of large amounts of data with the ease of use of relational databases. It enables advanced data operations like filtering, grouping, and aggregating, and the use of multi-column keys enhances the ability to link data across tables (aka joins).\n",
    "\n",
    "In the example below we create a table with camera images and bounding boxes, only for the frontal camera (camera_name.FRONT = 1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "X9KCT9YRUTPv",
    "outputId": "db12720c-4141-4733-d36b-519a9b5433b9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded image (387998 bytes) for camera_image.key.camera_name=1 1552440195362591 camera_image.key.camera_name=1\n",
      "Loaded bounding box for camera_box.key.camera_object_id='02f3a769-021f-49b0-b954-18e4fa1d5fde' camera_box.box=BoxAxisAligned2d(center=Vec2d(x=900.01575, y=665.69586), size=Vec2d(x=13.894980000000032, y=22.737240000000043))\n"
     ]
    }
   ],
   "source": [
    "camera_image_df = read('camera_image')\n",
    "# Filter the images from camera=1\n",
    "# NOTE: We could also use push down filters while reading the parquet files as well\n",
    "# Details https://docs.dask.org/en/stable/generated/dask.dataframe.read_parquet.html#dask.dataframe.read_parquet\n",
    "camera_image_df = camera_image_df[camera_image_df['key.camera_name'] == 1]\n",
    "\n",
    "camera_box_df = read('camera_box')\n",
    "# Inner join the camera_image table with the camera_box table.\n",
    "df = camera_image_df.merge(\n",
    "    camera_box_df,\n",
    "    on=[\n",
    "        'key.segment_context_name',\n",
    "        'key.frame_timestamp_micros',\n",
    "        'key.camera_name',\n",
    "    ],\n",
    "    how='inner',\n",
    ")\n",
    "\n",
    "# Create corresponding components from the raw\n",
    "_, row = next(iter(df.iterrows()))\n",
    "\n",
    "camera_image = v2.CameraImageComponent.from_dict(row)\n",
    "camera_box = v2.CameraBoxComponent.from_dict(row)\n",
    "print(\n",
    "    f'Loaded image ({len(camera_image.image)} bytes) for'\n",
    "    f' {camera_image.key.camera_name=} {camera_image.key.frame_timestamp_micros} {camera_image.key.camera_name=}'\n",
    ")\n",
    "print(\n",
    "    'Loaded bounding box for'\n",
    "    f' {camera_box.key.camera_object_id=} {camera_box.box=}'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5LCOOzVRBL6K"
   },
   "source": [
    "Since we use the same convention to name key columns for all components we can automatically determine a subset of columns to JOIN two tables by calling `v2.merge` function (SQL dialects use the \"join\" term, while Pandas and Dask call it \"merge\"):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "height": 357
    },
    "id": "csAJkgO25Uau",
    "outputId": "7b84dd78-6885-4d19-c9f2-934057a48046"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-224db546-6e0f-4036-8096-cde59cf7451f\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key.segment_context_name</th>\n",
       "      <th>key.frame_timestamp_micros</th>\n",
       "      <th>key.camera_name</th>\n",
       "      <th>[CameraImageComponent].image</th>\n",
       "      <th>[CameraImageComponent].pose.transform</th>\n",
       "      <th>[CameraImageComponent].velocity.linear_velocity.x</th>\n",
       "      <th>[CameraImageComponent].velocity.linear_velocity.y</th>\n",
       "      <th>[CameraImageComponent].velocity.linear_velocity.z</th>\n",
       "      <th>[CameraImageComponent].velocity.angular_velocity.x</th>\n",
       "      <th>[CameraImageComponent].velocity.angular_velocity.y</th>\n",
       "      <th>[CameraImageComponent].velocity.angular_velocity.z</th>\n",
       "      <th>[CameraImageComponent].pose_timestamp</th>\n",
       "      <th>[CameraImageComponent].rolling_shutter_params.shutter</th>\n",
       "      <th>[CameraImageComponent].rolling_shutter_params.camera_trigger_time</th>\n",
       "      <th>[CameraImageComponent].rolling_shutter_params.camera_readout_done_time</th>\n",
       "      <th>key.camera_object_id</th>\n",
       "      <th>[CameraBoxComponent].box.center.x</th>\n",
       "      <th>[CameraBoxComponent].box.center.y</th>\n",
       "      <th>[CameraBoxComponent].box.size.x</th>\n",
       "      <th>[CameraBoxComponent].box.size.y</th>\n",
       "      <th>[CameraBoxComponent].type</th>\n",
       "      <th>[CameraBoxComponent].difficulty_level.detection</th>\n",
       "      <th>[CameraBoxComponent].difficulty_level.tracking</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10023947602400723454_1120_000_1140_000</td>\n",
       "      <td>1552440195362591</td>\n",
       "      <td>1</td>\n",
       "      <td>b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...</td>\n",
       "      <td>[0.6990588408049747, 0.7150334085392328, 0.006...</td>\n",
       "      <td>1.337475</td>\n",
       "      <td>-1.359002</td>\n",
       "      <td>0.027288</td>\n",
       "      <td>-0.015392</td>\n",
       "      <td>0.012218</td>\n",
       "      <td>0.011145</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.006981</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.0</td>\n",
       "      <td>02f3a769-021f-49b0-b954-18e4fa1d5fde</td>\n",
       "      <td>900.01575</td>\n",
       "      <td>665.69586</td>\n",
       "      <td>13.89498</td>\n",
       "      <td>22.73724</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10023947602400723454_1120_000_1140_000</td>\n",
       "      <td>1552440195362591</td>\n",
       "      <td>1</td>\n",
       "      <td>b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...</td>\n",
       "      <td>[0.6990588408049747, 0.7150334085392328, 0.006...</td>\n",
       "      <td>1.337475</td>\n",
       "      <td>-1.359002</td>\n",
       "      <td>0.027288</td>\n",
       "      <td>-0.015392</td>\n",
       "      <td>0.012218</td>\n",
       "      <td>0.011145</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.006981</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0611ea50-f652-406d-942a-9ec7e3e90546</td>\n",
       "      <td>1303.60176</td>\n",
       "      <td>697.90695</td>\n",
       "      <td>29.05314</td>\n",
       "      <td>54.94833</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10023947602400723454_1120_000_1140_000</td>\n",
       "      <td>1552440195362591</td>\n",
       "      <td>1</td>\n",
       "      <td>b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...</td>\n",
       "      <td>[0.6990588408049747, 0.7150334085392328, 0.006...</td>\n",
       "      <td>1.337475</td>\n",
       "      <td>-1.359002</td>\n",
       "      <td>0.027288</td>\n",
       "      <td>-0.015392</td>\n",
       "      <td>0.012218</td>\n",
       "      <td>0.011145</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.006981</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.0</td>\n",
       "      <td>08b1e19e-b912-4963-a7bb-55e138bbc25e</td>\n",
       "      <td>1828.45305</td>\n",
       "      <td>741.48666</td>\n",
       "      <td>67.58013</td>\n",
       "      <td>99.15963</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10023947602400723454_1120_000_1140_000</td>\n",
       "      <td>1552440195362591</td>\n",
       "      <td>1</td>\n",
       "      <td>b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...</td>\n",
       "      <td>[0.6990588408049747, 0.7150334085392328, 0.006...</td>\n",
       "      <td>1.337475</td>\n",
       "      <td>-1.359002</td>\n",
       "      <td>0.027288</td>\n",
       "      <td>-0.015392</td>\n",
       "      <td>0.012218</td>\n",
       "      <td>0.011145</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.006981</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11165fc3-e936-4722-9257-1574c6752eab</td>\n",
       "      <td>951.17454</td>\n",
       "      <td>658.74837</td>\n",
       "      <td>13.89498</td>\n",
       "      <td>15.78975</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10023947602400723454_1120_000_1140_000</td>\n",
       "      <td>1552440195362591</td>\n",
       "      <td>1</td>\n",
       "      <td>b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...</td>\n",
       "      <td>[0.6990588408049747, 0.7150334085392328, 0.006...</td>\n",
       "      <td>1.337475</td>\n",
       "      <td>-1.359002</td>\n",
       "      <td>0.027288</td>\n",
       "      <td>-0.015392</td>\n",
       "      <td>0.012218</td>\n",
       "      <td>0.011145</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.006981</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>1.552440e+09</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24508c28-3154-4b7f-964d-70b3ed7d5a9d</td>\n",
       "      <td>608.85276</td>\n",
       "      <td>715.59147</td>\n",
       "      <td>67.58013</td>\n",
       "      <td>99.15963</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-224db546-6e0f-4036-8096-cde59cf7451f')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-224db546-6e0f-4036-8096-cde59cf7451f button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-224db546-6e0f-4036-8096-cde59cf7451f');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                 key.segment_context_name  key.frame_timestamp_micros  key.camera_name                       [CameraImageComponent].image              [CameraImageComponent].pose.transform  ...  [CameraBoxComponent].box.size.x  [CameraBoxComponent].box.size.y  [CameraBoxComponent].type  [CameraBoxComponent].difficulty_level.detection  [CameraBoxComponent].difficulty_level.tracking\n",
       "0  10023947602400723454_1120_000_1140_000            1552440195362591                1  b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...  [0.6990588408049747, 0.7150334085392328, 0.006...  ...                         13.89498                         22.73724                          1                                              2.0                                             2.0\n",
       "1  10023947602400723454_1120_000_1140_000            1552440195362591                1  b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...  [0.6990588408049747, 0.7150334085392328, 0.006...  ...                         29.05314                         54.94833                          2                                              NaN                                             NaN\n",
       "2  10023947602400723454_1120_000_1140_000            1552440195362591                1  b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...  [0.6990588408049747, 0.7150334085392328, 0.006...  ...                         67.58013                         99.15963                          2                                              NaN                                             NaN\n",
       "3  10023947602400723454_1120_000_1140_000            1552440195362591                1  b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...  [0.6990588408049747, 0.7150334085392328, 0.006...  ...                         13.89498                         15.78975                          1                                              2.0                                             2.0\n",
       "4  10023947602400723454_1120_000_1140_000            1552440195362591                1  b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00...  [0.6990588408049747, 0.7150334085392328, 0.006...  ...                         67.58013                         99.15963                          2                                              NaN                                             NaN\n",
       "\n",
       "[5 rows x 24 columns]"
      ]
     },
     "execution_count": 0,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Merge\n",
    "df = v2.merge(read('camera_image'), read('camera_box'))\n",
    "\n",
    "# Show the list of columns in the combined DataFrame\n",
    "df.head()"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "3tW2MkdrIn6F",
    "IWR8_hets6c7"
   ],
   "name": "tutorial_v2.ipynb",
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}