Validation/refcat_loader_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1c8c66c7",
   "metadata": {},
   "source": [
    "# Refcat Loader Demo\n",
    "\n",
    "<br>Owner: **Keith Bechtol** ([@bechtol](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@bechtol))\n",
    "<br>Last Verified to Run: **2021-07-09**\n",
    "<br>Verified Stack Release: **w_2021_25**\n",
    "\n",
    "This notebook demonstrates how to load a reference catalog with color terms applied. Thanks to Eli Rykoff for adding this functionality.\n",
    "\n",
    "This notebook uses HSC RC2 dataset (a few tracts of HSC data that are reprocessed ~monthly for routine science performance evaluation of the science pipelines).\n",
    "\n",
    "### Learning Objectives\n",
    "After working through and studying this notebook you should be able to\n",
    "   1. Access the schema of `sourceTable_visit` catalog \n",
    "   2. Load `sourceTable_visit` catalog into memory, including subset of columns (reading from a parquet file)\n",
    "   3. Load subset of reference catalog that overlaps the same region of the sky.\n",
    "\n",
    "### Logistics\n",
    "This notebook is intended to be runnable on **only** on `lsst-lsp-stable.ncsa.illinois.edu` from a local git clone of the [StackClub](https://github.com/LSSTScienceCollaborations/StackClub) repo."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8990f529",
   "metadata": {},
   "source": [
    "## Setup\n",
    "You can find the Stack version by using `eups list -s` on the terminal command line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3af969a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Site, host, and stack version\n",
    "! echo $EXTERNAL_INSTANCE_URL\n",
    "! echo $HOSTNAME\n",
    "! eups list -s | grep lsst_distrib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45f2d0a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, os.path\n",
    "import numpy as np\n",
    "from astropy.time import Time\n",
    "\n",
    "import lsst.geom\n",
    "from lsst.pipe.tasks.loadReferenceCatalog import LoadReferenceCatalogConfig, LoadReferenceCatalogTask\n",
    "from lsst.meas.algorithms import ReferenceObjectLoader\n",
    "import lsst.daf.butler as dafButler\n",
    "from lsst.utils import getPackageDir\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib widget"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c99d968",
   "metadata": {},
   "source": [
    "## Explore the `sourceTable_visit` and `objectTable_tract` Tables\n",
    "\n",
    "The `sourceTable_visit` and `objectTable_tract` tables are stored as parquet files and it is possible to load just a subset of columns for rapid data access.\n",
    "\n",
    "First we need to set up the butler, in this case, pointing to a HSC RC2 dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c541a6fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "repo = '/repo/main/'\n",
    "config= os.path.join(repo,'butler.yaml')\n",
    "butler = dafButler.Butler(config=config)\n",
    "registry = butler.registry\n",
    "collections = ['HSC/runs/RC2/w_2021_18/DM-29973']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73a5be01",
   "metadata": {},
   "source": [
    "Access the column names for `sourceTable_visit`. The cell below uses [getDeferred](https://pipelines.lsst.io/py-api/lsst.daf.butler.Butler.html#lsst.daf.butler.Butler.getDeferred) syntax to return a `DeferredDatasetHandle` which can later retrieve a dataset, after an immediate registry lookup. In this case, we don't need the catalog itself -- we just want the columns. We will use `getDeferred` again when accessing the reference catalogs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5c1d3be",
   "metadata": {},
   "outputs": [],
   "source": [
    "dat_refs_source_table = sorted(registry.queryDatasets('sourceTable_visit', collections=collections))\n",
    "butler.getDeferred('sourceTable_visit', dat_refs_source_table[0].dataId, collections=collections).get(component='columns').values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f370c25",
   "metadata": {},
   "source": [
    "Similarly, we can access the column names for `objectTable_tract`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a23a38d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dat_refs_object_table = sorted(registry.queryDatasets('objectTable_tract', collections=collections))\n",
    "butler.getDeferred('objectTable_tract', dat_refs_object_table[0].dataId, collections=collections).get(component='columns').values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "220b4444",
   "metadata": {},
   "source": [
    "Load all columns for all sources in a visit..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a868e81",
   "metadata": {},
   "outputs": [],
   "source": [
    "catalog = butler.getDirect(dat_refs_source_table[0])\n",
    "catalog"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a83ad6a",
   "metadata": {},
   "source": [
    "Or load just a few columns of interest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea69586a",
   "metadata": {},
   "outputs": [],
   "source": [
    "catalog = butler.getDirect(dat_refs_source_table[0], parameters={\"columns\": ['sourceId', 'coord_ra', 'coord_dec']})\n",
    "catalog"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c6e40e6",
   "metadata": {},
   "source": [
    "Note that the `objectTable_tract` for HSC RC2 is large enough that one cannot load all columns into memory on the RSP, so one must specify a subset of columns. You may have to restart the kernel due to memory overflow if you make this mistake."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9223fca7",
   "metadata": {},
   "source": [
    "## Load Reference Catalog\n",
    "\n",
    "Next we demonstrate how to load a reference catalog, in this case, either Gaia or PS1 depending on whether you are more interested in astrometry (with proper motions) or photometry (with color terms)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43cf31dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Toggle between\n",
    "# refDataset = 'gaia_dr2_20200414'\n",
    "refDataset = 'ps1_pv3_3pi_20170110'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d5721d1",
   "metadata": {},
   "source": [
    "Set up butler:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5154fdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "repo = '/repo/main/'\n",
    "config = os.path.join(repo, 'butler.yaml')\n",
    "butler = dafButler.Butler(config=config)\n",
    "registry = butler.registry\n",
    "collection = 'refcats'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c22343cf",
   "metadata": {},
   "source": [
    "Let's see what reference catalogs are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3607bb45",
   "metadata": {},
   "outputs": [],
   "source": [
    "registry.getCollectionSummary(collection).datasetTypes.names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1499fb50",
   "metadata": {},
   "source": [
    "The first step to load reference catalogs is to select a specific region of the sky because we don't want to load the entire catalog into memory at once. Fortunately, the reference catalogs are spatially sharded so that we load a subset of the full reference catalog that is spatially localized. In this example, we first access the dataset references for shards that spatially overlap one of the HSC visits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68a10c5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "visit = 35892\n",
    "datasetRefs = list(registry.queryDatasets(datasetType=refDataset,\n",
    "                                          collections=collection,\n",
    "                                          instrument='HSC',\n",
    "                                          skymap='hsc_rings_v1',\n",
    "                                          where=f'visit={visit}').expanded())\n",
    "\n",
    "dataIds = [_.dataId for _ in datasetRefs]\n",
    "\n",
    "# Get DeferredDatasetHandles for reference catalog\n",
    "refCats = [butler.getDeferred(refDataset, _, collections=['refcats'])\n",
    "           for _ in dataIds]\n",
    "\n",
    "cat_ref_example = butler.getDirect(datasetRefs[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8161cb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_ref_example.asAstropy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ef05f1b",
   "metadata": {},
   "source": [
    "Next we load the HSC source catalog for that visit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6203adb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the HSC catalog for comparsion\n",
    "refs = list(registry.queryDatasets(datasetType='sourceTable_visit',\n",
    "                                   collections=['HSC/runs/RC2/w_2021_18/DM-29973'],\n",
    "                                   instrument='HSC',\n",
    "                                   skymap='hsc_rings_v1',\n",
    "                                   where=f'visit={visit}'))\n",
    "cat_hsc = butler.getDirect(refs[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da764cbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_hsc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6e8b5ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "config = LoadReferenceCatalogConfig()\n",
    "config.refObjLoader.ref_dataset_name = refDataset\n",
    "\n",
    "if refDataset == 'gaia_dr2_20200414':\n",
    "    # Apply proper motions for Gaia catalog\n",
    "    config.refObjLoader.requireProperMotion = True\n",
    "    config.refObjLoader.anyFilterMapsToThis = 'phot_g_mean'\n",
    "    config.doApplyColorTerms = False\n",
    "else:\n",
    "    # Apply color terms for PS1 catalog\n",
    "    config.refObjLoader.load(os.path.join(getPackageDir('obs_subaru'),\n",
    "                                          'config',\n",
    "                                          'filterMap.py'))\n",
    "    config.colorterms.load(os.path.join(getPackageDir('obs_subaru'),\n",
    "                                        'config',\n",
    "                                        'colorterms.py'))\n",
    "\n",
    "# Set the epoch for proper motions. Here picking a random date:\n",
    "epoch = Time('2021-06-10')\n",
    "\n",
    "loaderTask = LoadReferenceCatalogTask(config=config,\n",
    "                                      dataIds=dataIds,\n",
    "                                      refCats=refCats)\n",
    "\n",
    "# Define center relative to HSC catalog\n",
    "center = lsst.geom.SpherePoint(np.median(cat_hsc['coord_ra']),\n",
    "                               np.median(cat_hsc['coord_dec']),\n",
    "                               lsst.geom.degrees)\n",
    "# Alternatively, define center relative to reference catalog\n",
    "# center = lsst.geom.SpherePoint(cat_ref_example['coord_ra'][0],\n",
    "#                                cat_ref_example['coord_dec'][0],\n",
    "#                                lsst.geom.radians)\n",
    "print('Using center (RA, DEC) =', center)\n",
    "\n",
    "cat_ref = loaderTask.getSkyCircleCatalog(center,\n",
    "                                         1.0*lsst.geom.degrees,\n",
    "                                         ['HSC-G', 'HSC-R'],\n",
    "                                         epoch=epoch)\n",
    "print('Found %i reference catalog objects'%(len(cat_ref)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14534f3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_ref"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a4ed5c8",
   "metadata": {},
   "source": [
    "Note that the reference catalog fluxes have been converted to magnitudes in the HSC system if color terms have been applied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a61a5939",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.scatter(cat_ref['ra'], cat_ref['dec'], marker='.', s=10, edgecolor='none', label='Reference')\n",
    "plt.scatter(cat_hsc['coord_ra'], cat_hsc['coord_dec'], marker='.', s=1, edgecolor='none', label='HSC')\n",
    "plt.xlabel('RA (deg)')\n",
    "plt.ylabel('Dec (deg)')\n",
    "plt.legend(markerscale=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d69263a3",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Perform a spatial matching between the HSC and reference catalog and compare the astrometry and photometry of matched objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3aef9a4",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}