DP02_04a_Introduction_to_the_Butler.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src = https://project.lsst.org/sites/default/files/Rubin-O-Logo_0.png width=250 style=\"padding: 10px\"> \n",
    "<br><b>Introduction to Data Access with the Butler</b> <br>\n",
    "Contact author(s): Alex Drlica-Wagner <br>\n",
    "Last verified to run: 2024-05-01 <br>\n",
    "LSST Science Pipelines version: Weekly 2024_16 <br>\n",
    "Container Size: large <br>\n",
    "Targeted learning level: beginner <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Description:** This notebook uses the Butler to query DP0 images and catalogs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Skills:** Query and retrieve images and catalog data with the Butler."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**LSST Data Products:** Images (calexp, goodSeeingDiff_differenceExp, deepCoadd) and catalogs (sourceTable, diaSourceTable, objectTable, forcedSourceTable, forcedSourceOnDiaObjectTable)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Packages:** lsst.daf.butler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Credit:** This tutorial was originally developed by Alex Drlica-Wagner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Get Support:**\n",
    "Find DP0-related documentation and resources at <a href=\"https://dp0.lsst.io\">dp0.lsst.io</a>. Questions are welcome as new topics in the <a href=\"https://community.lsst.org/c/support/dp0\">Support - Data Preview 0 Category</a> of the Rubin Community Forum. Rubin staff will respond to all questions posted there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction\n",
    "\n",
    "The Butler is the LSST Science Pipelines interface for managing, reading, and writing datasets. The Butler can be used to explore the contents of the DP0 data repository and access the DP0 data. The current version of the Butler (referred to as \"Gen3\") is still under development, and this notebook may be modified in the future. Full Butler documentation can be found [here](https://pipelines.lsst.io/modules/lsst.daf.butler/index.html).\n",
    "\n",
    "This notebook demonstrates how to:<br>\n",
    "1. Create an instance of the Butler<br>\n",
    "2. Use the Butler to retrieve image data<br>\n",
    "3. Use the Butler to retrieve catalog data<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Import packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import general python packages and several packages from the LSST Science Pipelines, including the Butler package and AFW Display, which will be used to display images.\n",
    "More details and techniques regarding image display with `AFW Display` can be found in the `rubin-dp0` GitHub Organization's [tutorial-notebooks](https://github.com/rubin-dp0/tutorial-notebooks) repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Generic python packages\n",
    "import pylab as plt\n",
    "import gc\n",
    "import numpy as np\n",
    "\n",
    "# LSST Science Pipelines (Stack) packages\n",
    "import lsst.daf.butler as dafButler\n",
    "import lsst.afw.display as afwDisplay\n",
    "\n",
    "# Set a standard figure size to use\n",
    "plt.rcParams['figure.figsize'] = (8.0, 8.0)\n",
    "afwDisplay.setDefaultBackend('matplotlib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create an instance of the Butler\n",
    "\n",
    "To create the Butler, we need to provide it with a configuration for the data set (often referred to as a \"data repository\").\n",
    "This configuration can be the path to a yaml file (often named `butler.yaml`), a directory path (in which case the Butler will look for a `butler.yaml` file in that directory), or a shorthand repository label (i.e., `dp02`). If no configuration is passed, the Butler will use a default value (this is not recommended in most cases).\n",
    "For the purposes of accessing the DP0.2 data, we will use the `dp02` label.\n",
    "\n",
    "In addition to the config, the Butler also takes a set of data `collections`. \n",
    "Collections are lightweight groups of self-consistent datasets such as raw images, calibration files, reference catalogs, and the outputs of a processing run.\n",
    "You can find out more about collections [here](https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/organizing.html#collections).\n",
    "For the purposes of accessing the DP0.2 data, we will use the `2.2i/runs/DP0.2` collection. \n",
    "Here, '2.2i' refers to the imSim run that was used to generated the simulated data and 'runs' refers to the fact that it is processed data.\n",
    "\n",
    "Create an instance of the Butler pointing to the DP0.2 repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "config = 'dp02'\n",
    "collections = '2.2i/runs/DP0.2'\n",
    "butler = dafButler.Butler(config, collections=collections)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learn more about the Butler by uncommenting the following line and executing the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# help(butler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Butler image access\n",
    "\n",
    "The Butler can be used to access DP0 image data products. The DP0.2 [Data Products Definition Document (DPDD)](https://dp0-2.lsst.io/data-products-dp0-2/index.html#images) describes three different types of image data products: processed visit images (PVIs; `calexp`),  difference images (`goodSeeingDiff_differenceExp`), and coadded images (`deepCoadd`). We will demonstrate how to access each of these.\n",
    "\n",
    "In order to access a data product through the Butler, we will need to tell the Butler what data we want to access. This call generally has two components: the `datasetType` tells the Butler what type of data product we are seeking (e.g., `deepCoadd`, `calexp`, `objectTable`), and the `dataId` is a dictionary-like identifier for a specific data product (more info on dataIds can be found [here](https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/dimensions.html#data-ids))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Processed Visit Images\n",
    "\n",
    "Processed visit images can be accessed with the `calexp` datasetType. \n",
    "These are image data products derived from processing of a single detector in a single visit of the LSST Camera. \n",
    "To access a `calexp`, the minimal information that we will need to provide is the visit number and the detector number. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "datasetType = 'calexp'\n",
    "dataId = {'visit': 192350, 'detector': 175}\n",
    "calexp = butler.get(datasetType, dataId=dataId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view all parameters that can be passed to the dataId for calexp, we can use the Butler registry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(butler.registry.getDatasetType(datasetType))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot a calexp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "display = afwDisplay.Display(frame=fig)\n",
    "display.scale('asinh', 'zscale')\n",
    "display.mtv(calexp.image)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "del calexp\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Difference Images\n",
    "\n",
    "Difference images can be accessed with the `goodSeeingDiff_differenceExp` datasetType. \n",
    "These are PVIs which have had a template image subtracted from them. \n",
    "These data products are used to measure time-varying difference in the fluxes of astronomical sources.\n",
    "To access a difference image we require the visit and detector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "datasetType = 'goodSeeingDiff_differenceExp'\n",
    "dataId = {'visit': 192350, 'detector': 175}\n",
    "diffexp = butler.get(datasetType, dataId=dataId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot a goodSeeingDiff_differenceExp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "display = afwDisplay.Display(frame=fig)\n",
    "display.scale('asinh', 'zscale')\n",
    "display.mtv(diffexp.image)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "del diffexp\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3. Coadded Images\n",
    "\n",
    "Coadded images combine multiple PVIs to assemble a deeper image; they can be accessed with the `deepCoadd` datasetType.\n",
    "Coadd images are divided into “tracts” (a spherical convex polygon) and tracts are divided into “patches” (a quadrilateral sub-region, with a size in pixels chosen to fit easily into memory on desktop computers). \n",
    "Coadd patches are roughly the same size as a single-detector `calexp` image.\n",
    "To access a `deepCoadd`, we need to specify the tract, patch, and band that we are interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "datasetType = 'deepCoadd'\n",
    "dataId = {'tract': 4431, 'patch': 17, 'band': 'i'}\n",
    "coadd = butler.get(datasetType, dataId=dataId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot a deepCoadd."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "display = afwDisplay.Display(frame=fig)\n",
    "display.scale('asinh', 'zscale')\n",
    "display.mtv(coadd.image)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "del coadd\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 4. Butler table access\n",
    "\n",
    "While the preferred technique to access DP0 catalogs is through the table access protocol (TAP) service, the Butler can also provide access to these data products. \n",
    "We will demonstrate how to access several different [catalog products described in the DPDD](https://dp0-2.lsst.io/data-products-dp0-2/index.html#catalogs). \n",
    "The catalogs are returned by the Butler as pandas DataFrame objects, which can be further manipulated by the user.\n",
    "The full descriptions of the column schema from the DP0.2 tables can be found [here](https://dm.lsst.org/sdm_schemas/browser/dp02.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1. Processed Visit Sources\n",
    "\n",
    "The `sourceTable` provides astrometric and photometric measurements for sources detected in the individual PVIs (`calexp`). \n",
    "Thus, to access the `sourceTable` for a specific PVI, we require similar information as was used to access the `calexp`.\n",
    "More information on the columns of the `sourceTable` can be found [here](https://dm.lsst.org/sdm_schemas/browser/dp02.html#Source)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "datasetType = 'sourceTable'\n",
    "dataId = {'visit': 192350, 'detector': 175}\n",
    "src = butler.get(datasetType, dataId=dataId)\n",
    "print(f\"Retrieved catalog of {len(src)} sources.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "src"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "del src\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4.2. Difference Image Sources\n",
    "\n",
    "The `diaSourceTable` provides astrometric and photometric measurements for sources detected in the difference images. \n",
    "To access the `diaSourceTable` for a specific difference image, we require similar information as was used to access the difference image itself.\n",
    "However, the `diaSourceTable` groups together all sources detected in a single visit, and thus the detector number is not used.\n",
    "More information on the columns of the `diaSourceTable` can be found [here](https://dm.lsst.org/sdm_schemas/browser/dp02.html#DiaSource)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "datasetType = 'diaSourceTable'\n",
    "dataId = {'visit': 192350}\n",
    "dia_src = butler.get(datasetType, dataId=dataId)\n",
    "print(f\"Retrieved catalog of {len(dia_src)} DIA sources.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dia_src"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To retrieve sources from a specific detector, we can select a subset of the sources returned by our query to the `diaSourceTable`. In particular, we note that the `ccdVisitId` column contains a value that combines the visit and detector IDs. We could build this value with some simple Python string formatting; however, the more robust way is to ask the Butler what this value should be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataId = {'visit': 192350, 'detector': 5}\n",
    "det_string = str(dataId['detector'])\n",
    "ccd_visit_id = np.longlong(str(dataId['visit'])+det_string.rjust(3, '0'))\n",
    "selection = dia_src['ccdVisitId'] == ccd_visit_id\n",
    "dia_src_ccd = dia_src[selection]\n",
    "print(f\"Found catalog of {len(dia_src_ccd)} DIA sources associated with {dataId}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the table selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dia_src_ccd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "del dia_src, selection, dia_src_ccd\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3. Coadd Objects\n",
    "\n",
    "The `objectTable` provides astrometric and photometric measurements for objects detected in coadded images. \n",
    "These objects are assembled across the set of coadd images in all bands, and thus contain multi-band photometry. \n",
    "For this reason, we do not specify the band in the `dataId`.\n",
    "More information on the columns of the `objectTable` can be found [here](https://dm.lsst.org/sdm_schemas/browser/dp02.html#Object)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "datasetType = 'objectTable'\n",
    "dataId = {'tract': 4431, 'patch': 17}\n",
    "obj = butler.get(datasetType, dataId=dataId)\n",
    "print(f\"Retrieved catalog of {len(obj)} objects.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "obj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "del obj\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4. Difference Image Objects\n",
    "\n",
    "The `diaObjectTable` contains derived summary parameters for DIA sources associated by sky location and includes lightcurve statistics (e.g., flux chi2, Stetson J).\n",
    "The DIA objects can be accessed from the `diaObjectTable_tract` table. \n",
    "As implied by the table name, it is intended to be queried by coadd tract.\n",
    "More information on the `diaObjectTable_tract` can be found [here](https://dm.lsst.org/sdm_schemas/browser/dp02.html#DiaObject)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "datasetType = 'diaObjectTable_tract'\n",
    "dataId = {'tract': 4431}\n",
    "dia_obj = butler.get(datasetType, dataId=dataId)\n",
    "print(f\"Retrieved catalog of {len(dia_obj)} DIA objects.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dia_obj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "del dia_obj\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4.5. Forced Photometry Sources\n",
    "\n",
    "Forced photometry refers to the process of fitting for a source at a specific location in an image regardless of whether the source has been detected in that image. \n",
    "This is useful when objects are not detected in all bands (e.g., drop outs) or observations (e.g., transient or variable objects). \n",
    "The `forcedSourceTable` contains forced photometry performed on the individual PVIs at the locations of all detected objects and linked to the `objectTable`.\n",
    "In contrast, the `forcedSourceOnDiaObjectTable` contains forced photometry on the individual PVIs at the locations of all objects in the `diaObjectTable`.\n",
    "Note that the tables returned by these butler queries are quite large and can fill up the memory available to your notebook.\n",
    "\n",
    "More information on the columns of the `forcedSourceTable` can be found [here](https://dm.lsst.org/sdm_schemas/browser/dp02.html#ForcedSource), and the columns of the `forcedSourceOnDiaObjectTable` are similar.\n",
    "\n",
    "> **Warning:** forcedSourceTable takes dataIds comprised of tract and patch only, which returns too many sources for a Jupyter Notebook with a small or medium container.\n",
    "\n",
    "Instead of forcedSourceTable, it is recommended to use forcedSource (which takes dataIds comprised of e.g., band, instrument, detector, and visit) and to apply spatial and temporal constraints whenever applicable.\n",
    "This is considered intermediate-level use of the Butler, and so is demonstrated in DP0.2 tutorial notebook 04b.\n",
    "\n",
    "**Only uncomment and execute the three cells below if you are using a large container.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# datasetType = 'forcedSourceTable'\n",
    "# dataId = {'tract': 4431, 'patch': 16, 'band':'i'}\n",
    "# forced_src = butler.get(datasetType, dataId=dataId)\n",
    "# print(f\"Retrieved catalog of {len(forced_src)} forced sources.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# forced_src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# del forced_src\n",
    "# gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Query the forcedSourceOnDiaObjectTable.\n",
    "This will execute with a medium container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "datasetType = 'forcedSourceOnDiaObjectTable'\n",
    "dataId = {'tract': 4431, 'patch': 16}\n",
    "dia_forced_src = butler.get(datasetType, dataId=dataId)\n",
    "print(f\"Retrieved catalog of {len(dia_forced_src)} DIA forced sources.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dia_forced_src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "del dia_forced_src\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Summary\n",
    "\n",
    "In this notebook we demonstrated Butler access for a set of image and catalog data products described in the [DPDD for DP0.2](https://dp0-2.lsst.io/data-products-dp0-2/index.html#dp0-2-data-products-definition-document-dpdd). \n",
    "However, we have not demonstrated the powerful capability of the Butler to query the holdings of a data repository.\n",
    "The full power of the Butler can be found in the [Butler documentation](https://pipelines.lsst.io/modules/lsst.daf.butler/index.html),\n",
    "or in DP0.2 tutorial notebook 04b \"Intermediate Butler Queries\"."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}