Validation/Ingest_and_load_local_refcat_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "87753239",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Ingest and load local refcat demo using DELVE_DR1\n",
    "\n",
    "<br>Owner: **Peter Ferguson** ([@psferguson](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@psferguson))\n",
    "<br>Last Verified to Run: **2022-04-01**\n",
    "<br>Verified Stack Release: **w_2021_49**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8aafaff",
   "metadata": {},
   "source": [
    "### Learning Objectives\n",
    "\n",
    "This notebook demonstrates how to: <br>\n",
    "1. Create an LSST-format reference catalog from an existing acii or fits reference catalog\n",
    "2. Create an empty Gen3 butler repo\n",
    "3. Ingest the LSST-format refcat into newly created repo\n",
    "4. Load the new reference catalog with the butler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b82073d2",
   "metadata": {},
   "source": [
    "### Set Up \n",
    "You can find the Stack version by using `eups list -s` on the terminal command line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31ccd1cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Site, host, and stack version\n",
    "! echo $EXTERNAL_INSTANCE_URL\n",
    "! echo $HOSTNAME\n",
    "! eups list -s | grep lsst_distrib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "800a53da",
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess\n",
    "import numpy as np\n",
    "import pylab as plt\n",
    "import lsst.geom\n",
    "import lsst.daf.butler  as dafButler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3bb747a",
   "metadata": {},
   "source": [
    "### Create a gen3 reference catalog\n",
    "\n",
    "For this example we will create a refcat from a DELVE (DEcam Local Volume Exploration survey [Website](https://delve-survey.github.io/)) DR1 healpixel located on NCSA, and import it into a gen3 repo. \n",
    "\n",
    "Following the developer instructions [piplines refcat documentation](https://pipelines.lsst.io/v/daily/modules/lsst.meas.algorithms/creating-a-reference-catalog.html)\n",
    "\n",
    "The first step is to use the `ConvertReferenceCatalogTask` along with a config we create for the conversion to create a catalog in LSST format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0b616da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting names\n",
    "refcatDir='custom_refcat_demo'\n",
    "configFile=\"ingestConfigOverride.cfg\"\n",
    "inputFile=\"/project/shared/data/delve_dr1/cat/cat_hpx_07798.fits\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ba06237-665f-4cc9-b037-c6565b54c991",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This notebook will only run if refcatDir doesn't exist\n",
    "if os.path.exists(refcatDir):\n",
    "    msg = f\"Please remove directory '{refcatDir}' to continue:\\n! rm -r {refcatDir}\"\n",
    "    raise Exception(msg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d528713",
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir {refcatDir}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eabd41f6-2226-4551-b3f0-59a068ff3763",
   "metadata": {},
   "source": [
    "Below is the set of configs used in creating this refcat\n",
    " 1. Since the refcat is in FITS format we retarget the file reader\n",
    " 2. It is required to give a name to this refcat, in this case 'delve_dr1'\n",
    " 3. We also need to specify ra, dec, mag, and mag_error columns\n",
    " 4. Finally we can give the config a list of extra columns to include in the refcat (e.g., a star/galaxy classifier)\n",
    " \n",
    "for this tutorial we will change the HTM depth to 4 to improve runtime but an HTM depth of 7 is default.\n",
    "\n",
    "The `%%writefile` cell magic writes the contents of the following cell to `configFile`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c511f560",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile {configFile}\n",
    "from lsst.meas.algorithms.readFitsCatalogTask import ReadFitsCatalogTask\n",
    "\n",
    "# Default is ReadTextCatalogTask\n",
    "config.file_reader.retarget(ReadFitsCatalogTask)\n",
    "\n",
    "# String to pass to the butler to retrieve persisted files.\n",
    "config.dataset_config.ref_dataset_name='delve_dr1'\n",
    "\n",
    "\n",
    "config.dataset_config.indexer.name='HTM'\n",
    "\n",
    "# Depth of the HTM tree to make.  Default is depth=7 which gives ~ 0.3 sq. deg. per trixel.\n",
    "# for this tutorial we will change the HTM depth to 4 to improve runtime \n",
    "config.dataset_config.indexer['HTM'].depth=4\n",
    "\n",
    "# Number of python processes to use when ingesting.\n",
    "config.n_processes=5\n",
    "\n",
    "# Name of RA column\n",
    "config.ra_name='RA'\n",
    "\n",
    "# Name of Dec column\n",
    "config.dec_name='DEC'\n",
    "\n",
    "# Name of column to use as an identifier (optional).\n",
    "config.id_name='QUICK_OBJECT_ID'\n",
    "\n",
    "# The values in the reference catalog are assumed to be in AB magnitudes. List of column names to use for\n",
    "# photometric information.  At least one entry is required.\n",
    "config.mag_column_list=['MAG_PSF_G', 'MAG_PSF_R','MAG_PSF_I', 'MAG_PSF_Z']\n",
    "\n",
    "# A map of magnitude column name (key) to magnitude error column (value).\n",
    "config.mag_err_column_map={'MAG_PSF_G':'MAGERR_PSF_G', 'MAG_PSF_R':'MAGERR_PSF_R','MAG_PSF_I':'MAGERR_PSF_I', 'MAG_PSF_Z':'MAGERR_PSF_Z'}\n",
    "\n",
    "# Names of extra columns to include \n",
    "config.extra_col_names=['SPREAD_MODEL_G','SPREAD_MODEL_R','SPREAD_MODEL_I','SPREAD_MODEL_Z',\n",
    "                        'SPREADERR_MODEL_G', 'SPREADERR_MODEL_R', 'SPREADERR_MODEL_I', 'SPREADERR_MODEL_Z',\n",
    "                        'EXTINCTION_G', 'EXTINCTION_R', 'EXTINCTION_I', 'EXTINCTION_Z']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "175e1562-1861-405b-8c1d-a0024193cc5b",
   "metadata": {},
   "source": [
    "### Convert Files to LSST format\n",
    "We then use the `convertReferenceCatalog` command line tool to ingest the catalog, this takes a bit of time to run. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ea6bf8a-362d-4d8b-b59f-11bd1e27f65c",
   "metadata": {},
   "outputs": [],
   "source": [
    "! convertReferenceCatalog {refcatDir} {configFile} {inputFile}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc7aa7d4-148e-4dc2-a7ec-5adb025aaff2",
   "metadata": {},
   "source": [
    "example output:\n",
    "\n",
    "    lsst.ConvertReferenceCatalogTask INFO: Creating 2048 file locks.\n",
    "    lsst.ConvertReferenceCatalogTask INFO: File locks created.\n",
    "    lsst.ConvertReferenceCatalogTask INFO: Completed 1 / 1 files: 100 % complete \n",
    "    Completed refcat conversion. Ingest the resulting files with the following commands, substituting the path to your butler repo for REPO:\n",
    "        butler register-dataset-type REPO delve_dr1 SimpleCatalog htm7\n",
    "        butler ingest-files -t direct REPO gaia_dr2 refcats custom_refcat_demo/filename_to_htm.ecsv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74647ca6",
   "metadata": {},
   "source": [
    "### Create a gen3 repo and load catalog into it\n",
    "We now have a LSST format refcat, for this demo we will create a new gen3 repo to ingest the refcat into. \n",
    "\n",
    "The creation of an empty repo needs a `butler.yaml` file. (e.g `./custom_refcat_demo/butler.yaml`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "144f3e4d-6b21-48e0-ad87-006e29735cb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create this file using notebook shell commands\n",
    "filepath = \"./custom_refcat_demo/butler.yaml\"\n",
    "filecontent = \"\"\"\n",
    "datastore:\n",
    "  cls: lsst.daf.butler.datastores.fileDatastore.FileDatastore \n",
    "  root: <butlerRoot>\n",
    "registry:\n",
    "  db: sqlite:///<butlerRoot>/test.sqlite3 \n",
    "\"\"\" \n",
    "! echo \"$filecontent\" > $filepath\n",
    "! cat $filepath"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00520101-8b1c-4fa5-8aa9-8448a6f40e78",
   "metadata": {},
   "source": [
    "Now we can run the `butler create` command line task to create a new repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a70dfd10-40bf-4b52-a253-01f95f12dbcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "repoName=\"test_repo_gen3\"\n",
    "! mkdir {refcatDir}/{repoName}\n",
    "! touch {refcatDir}/{repoName}/test.sqlite3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f773827-4e50-418e-9add-8ab9efba9ab9",
   "metadata": {},
   "outputs": [],
   "source": [
    "!butler create {refcatDir}/{repoName} --seed-config {refcatDir}/butler.yaml --override"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa43e044-2a94-47bb-abe1-df2fddd37836",
   "metadata": {},
   "source": [
    "Now that we have an empty gen3 repo we can ingest the catalog into it with the following commands. \n",
    "\n",
    "    butler register-dataset-type REPO RefcatName SimpleCatalog htm3\n",
    "    butler ingest-files -t direct REPO RefcatName collectionName filename_to_htm.ecsv\n",
    "note the collection name must be a RUN not CHAIN type collection.\n",
    "\n",
    "Also note that the htm option (htm4) must match with the ingestion config. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "599d3703-7dec-47f9-af49-dfdb922b26e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!butler register-dataset-type {refcatDir}/{repoName} delve_dr1 SimpleCatalog htm4\n",
    "!butler ingest-files -t direct {refcatDir}/{repoName} delve_dr1 refcats {refcatDir}/filename_to_htm.ecsv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94664e83-bcc5-49ab-ac2a-55f28cce6493",
   "metadata": {},
   "source": [
    "### Loading the new refcat\n",
    "We can now load this new repo, and check the \"refcats\" collection to see what it contains. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2939111a",
   "metadata": {},
   "outputs": [],
   "source": [
    "butler = dafButler.Butler(refcatDir+\"/\"+repoName, writeable=True)\n",
    "registry = butler.registry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b254544-5cb0-4c4e-af83-b11e84dd0e92",
   "metadata": {},
   "outputs": [],
   "source": [
    "[i for i in list(registry.queryCollections())]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb2f30a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "registry.getCollectionSummary('refcats').datasetTypes.names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36357fe4-0df7-4775-95ec-9f3350cc9314",
   "metadata": {},
   "source": [
    "We can set a docstring for this refcat collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47a16f42-2129-4c5a-b8eb-90cc6940279a",
   "metadata": {},
   "outputs": [],
   "source": [
    "registry.setCollectionDocumentation('refcats', \"doc for delve dr1 refcat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f37e78c1-9608-4e13-83bb-829bc29e9cb4",
   "metadata": {},
   "source": [
    "No longer need this butler repo to be writeable "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d4713cc-3c8d-4bcf-841b-4db663d5350a",
   "metadata": {},
   "outputs": [],
   "source": [
    "butler = dafButler.Butler(refcatDir+\"/\"+repoName, writeable=False)\n",
    "registry = butler.registry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfb61aa1-26c9-4ef7-b406-5b05ece03409",
   "metadata": {},
   "outputs": [],
   "source": [
    "registry.getCollectionDocumentation('refcats')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4a45f37",
   "metadata": {},
   "outputs": [],
   "source": [
    "refDataset=\"delve_dr1\"\n",
    "refcatRefs = list(registry.queryDatasets(datasetType=refDataset,\n",
    "                                          collections=[\"refcats\"]).expanded())\n",
    "refDataIds=[_.dataId for _ in refcatRefs]\n",
    "refCatsDef = [butler.getDeferred(refDataset, __, collections=['refcats']) for __ in refDataIds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05ee2dbb-75d3-46a5-8970-9ae1c9ddfebc",
   "metadata": {},
   "outputs": [],
   "source": [
    "refCats=[butler.getDirect(__) for __ in refcatRefs]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "851666e5-2c9a-4fd7-85f0-4afd0fb9f884",
   "metadata": {},
   "source": [
    "Finally we can plot the loaded refcat. The two different colors arise because this DELVE DR1 healpixel has been sharded into two htm4 pixels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b733852b-1fb8-4ae8-b1d0-25b6098b8f87",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pylab as plt\n",
    "fit,ax=plt.subplots()\n",
    "for refCat in refCats:\n",
    "    ax.scatter(refCat[\"coord_ra\"], refCat[\"coord_dec\"], label=\"refcat\",s=0.01)\n",
    "plt.xlabel(\"RA\")\n",
    "plt.ylabel(\"DEC\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}