Basics/Calexp_guided_tour.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "# A Guided Tour of LSST Processed Visit Images (calexp)\n",
    "<br>Author(s): **David Shupe** ([@stargaser](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@stargaser))\n",
    "<br>Maintainer(s): **Douglas Tucker** ([@douglasleetucker](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@douglasleetucker))\n",
    "<br>Level: **Introductory**\n",
    "<br>Last Verified to Run: **2021-09-11**\n",
    "<br>Verified Stack Release: **w_2021_33**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "### Learning Objectives:\n",
    "\n",
    "This notebook investigates LSST stack processed visit images, conventionally referred to as `calexp` objects. The `calexp` is a Butler dataset type that corresponds to a single processed, calibrated, and characterized CCD and the metadata associated with that image (mask plane, psf, etc.). We will explore the content of the `calexp` object and then compare to coadd images.\n",
    "\n",
    "After working through this tutorial you should be able to:\n",
    "1. Investigate the content of a calexp (and to some extent, other Stack objects).\n",
    "2. Follow some best practices when working with `calexp` objects.\n",
    "\n",
    "### Logistics\n",
    "This notebook is intended to be run at `lsst-lsp-stable.ncsa.illinois.edu` or `data.lsst.cloud` from a local git clone of the [StackClub](https://github.com/LSSTScienceCollaborations/StackClub) repo.\n",
    "\n",
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Site, host, and stack version\n",
    "! echo $EXTERNAL_INSTANCE_URL\n",
    "! echo $HOSTNAME\n",
    "! eups list -s | grep lsst_distrib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# General imports\n",
    "import os\n",
    "from pprint import pprint\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stack imports\n",
    "import lsst.daf.butler   as dafButler        #load the Butler to access data \n",
    "import lsst.afw.display  as afwDisplay       #load lsst.afw.display to gain access to image visualization routines.  \n",
    "afwDisplay.setDefaultBackend('matplotlib')   #set the default display backend to matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieving and inspecting images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the DESC DC2 Run2.2i data set (which comprises a simulated 300 sq deg patch of 5 years of Rubin LSST WFD data). Here, we create a Butler for the data set we wish to examine and grab the calexp image we are interested in exploring. More information on the Butler can be found in other Stack Club tutorials.\n",
    "\n",
    "In the first part of the tutorial, we will examine the calibrated exposure (`calexp`) for a single `visit`.  In the second part of the tutorial we will examine the calibrated image for a coadd `patch`. In the following cell, we grab both of these data objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Location of the DC2 Gen3 repository on this site\n",
    "URL = os.getenv('EXTERNAL_INSTANCE_URL')\n",
    "if URL.endswith('data.lsst.cloud'): # IDF\n",
    "    repo = \"s3://butler-us-central1-dp01\"\n",
    "elif URL.endswith('ncsa.illinois.edu'): # NCSA\n",
    "    repo = \"/repo/dc2\"\n",
    "else:\n",
    "    raise Exception(f\"Unrecognized URL: {URL}\")\n",
    "\n",
    "collection='2.2i/runs/DP0.1'\n",
    "\n",
    "# Create the butler\n",
    "butler = dafButler.Butler(repo,collections=collection)\n",
    "registry = butler.registry        \n",
    "\n",
    "# Grab a calexp of interest\n",
    "dataId = {'visit': '512055', 'detector': 76, 'band':'i'}\n",
    "calexp = butler.get('calexp', **dataId)\n",
    "\n",
    "## Grab a deepCoadd tract and patch\n",
    "coadd_dataId = {'band':'i', 'skymap': 'DC2', 'tract': 4851, 'patch': 29}\n",
    "coadd = butler.get('deepCoadd', **coadd_dataId)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image planes / pixel data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In terms of pixel data, a calexp contains an image, a mask, and a variance.\n",
    "\n",
    "Let's see how to access the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp.image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `calexp.image` is a Stack object of type `ImageF` (an image with floating point values)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `calexp.image` object can be displayed directly using other tools in the LSST Stack. To show the pixel data, we will make use of the matplotlib backend to `lsst.afw.display`.\n",
    "\n",
    "Due to current limitations of this backend, the display must be defined and used in the same code cell, much as matplotlib commands in a notebook must all be in one cell to produce a plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the entire calexp is displayed, masks will be overlaid. Here we will eschew the mask display by showing only the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display = afwDisplay.Display()\n",
    "display.scale(\"asinh\", \"zscale\")\n",
    "display.mtv(calexp.image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also access the low-level pixel values as a `numpy` array, using the `.array` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = calexp.image.array\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.__class__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's list all the methods for our calexp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp_methods = [m for m in dir(calexp) if not m.startswith('_')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp_methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `calexp` also contains a higher level `MaskedImage` object, which combines both the pixel values and the integer mask assigned to them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp.maskedImage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `MaskedImage` can also be plotted directly by lsst.afw.display. More information can be found in the StackClub notebook on afw.display."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access the variance object and the underlying Numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp.variance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp.variance.array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access the mask and its underlying array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp.mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp.mask.array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the dimensions of the image, mask and variance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp.getDimensions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The image, maskedImage and Exposure objects in `lsst.afw.display` include information on **LSST pixels**, which are 0-based with an optional offset.\n",
    "\n",
    "For a calexp these are usually zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp.getXY0()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp.getX0(), calexp.getY0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access the wcs object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wcs = calexp.getWcs()\n",
    "wcs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The WCS object can be used e.g. to convert pixel coordinates into sky coordinates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wcs.pixelToSky(100.0, 100.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try accessing the metadata, and see what (header) keywords we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = calexp.getMetadata()\n",
    "pprint(metadata.toDict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#metadata.get('CCDTEMP')\n",
    "metadata.get('TELESCOP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Better metadata: ExposureInfo and VisitInfo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For many purposes, information about an exposure is obtainable via the ExposureInfo and VisitInfo classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp_info = calexp.getInfo()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visit_info = calexp_info.getVisitInfo()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[m for m in dir(visit_info) if not m.startswith('_')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obtain weather information for this visit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visit_info.getWeather()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if this calexp has a valid polygon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp_info.hasValidPolygon()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no valid polygon for the `calexp`; so the `polygon` variable in the next cell returns `None`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "polygon = calexp_info.getValidPolygon()\n",
    "print(polygon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The calexp is not a coadd so this method returns False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp_info.hasCoaddInputs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does the calexp contain transmission curve information?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp_info.hasTransmissionCurve()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does the calexp contain a World Coordinate System?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp_info.hasWcs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does the calexp have a detector?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp_info.hasDetector()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[m for m in dir(calexp_info.getDetector()) if not m.startswith('_')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does the calexp have an aperture correction map?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp_info.hasApCorrMap()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the aperture correction map and print some information about it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "apCorrMap = calexp_info.getApCorrMap()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k in apCorrMap.keys():\n",
    "    print(k, apCorrMap.get(k))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image PSF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if our calexp has a PSF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp.hasPsf()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "psf = calexp.getPsf()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The PSF object can be used to get a realization of a PSF at a specific point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lsst.geom import PointD\n",
    "psfimage = psf.computeImage(PointD(100.,100.))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the PSF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display = afwDisplay.Display()\n",
    "display.scale('asinh', min=0.0, max=1.e-3, unit='absolute')\n",
    "display.mtv(psfimage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access the calibration object which can be used to convert instrumental magnitudes to AB magnitudes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calib = calexp.getPhotoCalib()\n",
    "calib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image cutouts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make a cutout from the calexp in our session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lsst.geom      as afwGeom\n",
    "import lsst.afw.image as afwImage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbox = afwGeom.Box2I()\n",
    "bbox.include(afwGeom.Point2I(400,1400))\n",
    "bbox.include(afwGeom.Point2I(600,1600))\n",
    "\n",
    "cutout = calexp[bbox]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that when the image is displayed, the pixel values relate to the parent image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display = afwDisplay.Display()\n",
    "display.scale('asinh', 'zscale')\n",
    "display.mtv(cutout.image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The coordinate of the lower-left-hand pixel is XY0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cutout.getXY0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a cutout was all that was desired from the start, we could have used our BoundingBox together with our Butler to have read in only the cutout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cutout_calexp = butler.get('calexp', parameters={'bbox':bbox}, dataId=dataId)\n",
    "cutout_calexp.getDimensions()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display = afwDisplay.Display()\n",
    "display.scale('asinh', 'zscale')\n",
    "display.mtv(cutout_calexp.image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `clone` method makes a deep copy. The result can be sliced with a BoundingBox"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clone_cutout = calexp.clone()[bbox]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display = afwDisplay.Display()\n",
    "display.scale(\"asinh\", \"zscale\")\n",
    "display.mtv(clone_cutout.image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## Repeat for a coadd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this last section, we will look at a coadd image. Earlier, we used the Butler to retrieve both calexp and coadd images.  Let's see here what methods it provides:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coadd_methods = [m for m in dir(coadd) if not m.startswith('_')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coadd_methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set(coadd_methods).symmetric_difference(set(calexp_methods))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of the `set` command above shows that a calexp and a coadd have the same methods. This is expected, because they are instances of the same class (`ExposureF`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"calexp class: {calexp.__class__}\")\n",
    "print(f\"coadd  class: {coadd.__class__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `deepCoadd` and a visit `calexp` differ mainly in the masks, xy0 values, and depth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"calexp XY0: {calexp.getXY0()}\")\n",
    "calexp.mask.getMaskPlaneDict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"coadd XY0: {coadd.getXY0()}\")\n",
    "coadd.mask.getMaskPlaneDict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the coadd image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display = afwDisplay.Display()\n",
    "display.scale(\"asinh\", \"zscale\")\n",
    "display.mtv(coadd.image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}