Basics/afw_table_guided_tour.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# afwTables: A Guided Tour\n",
    "<br>Author(s): **Imran Hasan** ([@ih64](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@ih64))\n",
    "<br>Maintainer(s): **Douglas Tucker** ([@douglasleetucker](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@douglasleetucker))\n",
    "<br>Level: **Introductory**\n",
    "<br>Last Verified to Run: **2021-09-03**\n",
    "<br>Verified Stack Release: **w_2021_33**\n",
    "\n",
    "Catalogs of astronomical objects and their many  measurements will be a primary data product that LSST provides. Queries of those catalogs will be the starting point for almost all LSST science analyses. On the way to filling the LSST database with these catalogs, the science pipelines will generate and manipulate a lot of internal tables; the python class that the Stack defines and uses for these tables is called an \"afwTable\". \n",
    "\n",
    "### Learning Objectives:\n",
    "\n",
    "After working through this tutorial you should be able to: \n",
    "1. Make a bare bones afw schema and table;\n",
    "2. Set and get values in a schema and table;\n",
    "3. Navigate large schemas;\n",
    "4. Read and write a source detection catalog table;\n",
    "5. Learn to use source detection catalog methods, and to avoid common pitfalls;\n",
    "6. Learn to use source match vectors.\n",
    "\n",
    "### Logistics\n",
    "This notebook is intended to be run at `lsst-lsp-stable.ncsa.illinois.edu` or `data.lsst.cloud` from a local git clone of the [StackClub](https://github.com/LSSTScienceCollaborations/StackClub) repo.\n",
    "\n",
    "### Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Site, host, and stack version\n",
    "! echo $EXTERNAL_INSTANCE_URL\n",
    "! echo $HOSTNAME\n",
    "! eups list -s | grep lsst_distrib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll need the `stackclub` package to be installed. If you are not developing this package, you can install it using `pip`, like this:\n",
    "```\n",
    "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
    "```\n",
    "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! cd .. && python setup.py -q develop --user && cd -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When editing the `stackclub` package files, we want the latest version to be imported when we re-run the import command. To enable this, we need the %autoreload magic command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For this tutorial we'll need the following modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "#%matplotlib ipympl\n",
    "\n",
    "import os\n",
    "import warnings\n",
    "import numpy as np\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('seaborn-poster')\n",
    "from IPython.display import IFrame, display, Markdown\n",
    "plt.style.use('seaborn-talk')\n",
    "from astropy.io import ascii"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lsst.daf.butler as dafButler\n",
    "import lsst.daf.base as dafBase\n",
    "import lsst.geom\n",
    "import lsst.afw.table as afwTable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Your first table\n",
    "\n",
    "To begin, we will make a bare-bones afw table so that we can clearly showcase some important concepts. First we will make the simplest possible table, by hand. While creating tables by hand will not likely be the standard use case, it is useful from a tutorial standpoint, as it will allow us to excercise some concepts one at a time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# afw tables need a schema to tell the table how its data are organized\n",
    "# Lets have a look at a simple schema:\n",
    "min_schema = afwTable.SourceTable.makeMinimalSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# But what is the schema exactly? Printing it out can be informative\n",
    "print(min_schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our schema contains 4 Fields: one for each celestial coordinate, an id that uniquely defines it, and a 'parent', which lists the id of the source this source was deblended from. We will deal with the parent column in more detail in a few cells, but for now you can ignore it.\n",
    "\n",
    "Each field has some accompanying information to go along with it. In addition to its name, we get a helpful docstring describing it. We also get the units that values for this field must have. For example, any value associated with the id key has to be a long integer, and all entries for celestial coordniates have to be instances of an Angle class. We will showcase the Angle class shortly.\n",
    "\n",
    "If printing out the schema gives you more information that you want, you can get the names. If the names are informative enough, this might be all you need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_schema.getNames()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can also add another field to the schema, using a call pattern like this:\n",
    "min_schema.addField(\"r_mag\", type=np.float32, doc=\"r band flux\", units=\"mag\")\n",
    "# Lets make sure the field was added by printing out the schema once more:\n",
    "print(min_schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also ask for a ordered list of names. In this case where our schema contains only 4 names, there is not a clear advantage. However, we will soon encounter schemas that contain many dozens of names. Sifting through a ordered list of these may be preferable. The ordering here is not alphabetical, but instead mirrors the ordering of these fields in the schema. You can check that is the case by comparing the ordering of output in the next cell to the one above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_schema.getOrderedNames()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We pause here to point out some caveats. \n",
    "1. Schemas are append only. You can add new fields, but you cannot remove them. \n",
    "2. The units you use have to be understood by astropy. You can find a list of acceptable units at the bottom of this page http://docs.astropy.org/en/stable/units/index.html#module-astropy.units\n",
    "3. Specific types are allowed. The short and long of it is you may use floats, ints, longs, strings, Angle objects, and arrays. For more details you can go to the bottom of this page http://doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc/afw_table.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a schema, we can use it to make a table.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_table = afwTable.BaseCatalog(min_schema)\n",
    "# our table is empty, and we can check this by looking at its length\n",
    "print('our minimal table has {} rows'.format(len(min_table)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will add some data to our minimal catalog. Catalogs are collections of 'records', which themselves contain data. Therefore, we must first create records, and hand those records over in turn to our Table. Records must adhere to the schema that the Table has, and so we must add data in field by field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make a new record.\n",
    "rec = min_table.addNew()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# grab a hold of the keys for the record. We will use these to add data \n",
    "field_dict = min_schema.extract('*') #this returns a dictionary of all the fields\n",
    "field_dict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# access the dictionary one field at a time, and grab each field's key. \n",
    "# note these are instances of a Key object, and not to be confused with python dictionary keys.\n",
    "id_key = field_dict['id'].key\n",
    "ra_key = field_dict['coord_ra'].key\n",
    "dec_key = field_dict['coord_dec'].key\n",
    "parent_key = field_dict['parent'].key\n",
    "r_mag_key = field_dict['r_mag'].key\n",
    "\n",
    "#use the keys to add data in our record\n",
    "rec.set(id_key, 1)\n",
    "rec.set(r_mag_key, 19.0)\n",
    "rec.set(ra_key, lsst.geom.Angle(0.2, units=lsst.geom.radians))\n",
    "rec.set(dec_key, lsst.geom.Angle(-3.14, units=lsst.geom.radians))\n",
    "rec.set(parent_key, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice to set the ra and dec, we needed to create `geom.Angle` objects for them. The object contains both the value of the angle, and the units the angle is in. The units keyword is set to radians by default, and all the DM code works on radians internally. To keep consistency with this, it is largely considered good pratice to work in radians too, setting the keyword for clarity. \n",
    "\n",
    "If you insisted that the angle be in other units, you can set them using the the units keyword. Other typical choices are degrees, arcminutes, arcseconds. You can learn more about `lsst.geom.Angle` objects [here]( http://doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc/classlsst_1_1geom_1_1_angle.html)\n",
    "\n",
    "Additionally, we set the parent to zero. This means this record refers to the object before any deblending occoured. Lets look at our table now to see how it stands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will flesh out the parent column a bit more by adding our next record. Notice we can keep using the keys we defined above. Also notice our second record's parent is listed as 1. This means the object 2 was the result of being deblended from object 1, i.e. object 2 is a child object of object 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec = min_table.addNew()\n",
    "rec.set(id_key, 2)\n",
    "rec.set(r_mag_key, 18.5)\n",
    "rec.set(ra_key, lsst.geom.Angle(3.14, units=lsst.geom.radians))\n",
    "rec.set(dec_key, lsst.geom.Angle(2.0, units=lsst.geom.radians))\n",
    "rec.set(parent_key, 1)\n",
    "min_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One more caveat to note: in the output in the cell above, the table prints coordinates in radians by default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your turn. add one more record to our table\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now that we have multiple records in our table, we can select particular ones\n",
    "# tables support indexing\n",
    "min_table[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you may iterate over them too\n",
    "for rec in min_table:\n",
    "    print(rec.get(id_key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can grab values from particular records by using our schema keys\n",
    "min_table[1].get(id_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using source catalogs produced by DM\n",
    "\n",
    "A more typical use case will be to read in a catalog that is produced by a DM process.  We will show how to read in and work with a source catalog produced from the Data Management Stack in the following section. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data access\n",
    "If you know the path to your source catalog, there is a quick way to read it in. However, it is often more powerful to use the 'data butler' to fetch data for you. The butler knows about camera geometry, sensor characteristics, where data are located, and so forth. Having this anciliary information on hand is often very useful. For completeness we will demonstrate both ways of reading in a source catalog, with the note that it is largely considered better practice to use the data butler. \n",
    "\n",
    "The data butler has its own tutorial(s), and so we will defer further details on it until later. For now, you may think of it as an abstraction that allows you to quickly fetch catalogs for you. The user just needs to point the butler to where to look and what to look for.\n",
    "\n",
    "We use the DESC DC2 data set as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Location of the DC2 Gen3 repository on this site\n",
    "URL = os.getenv('EXTERNAL_INSTANCE_URL')\n",
    "if URL.endswith('data.lsst.cloud'): # IDF\n",
    "    repo = \"s3://butler-us-central1-dp01\"\n",
    "elif URL.endswith('ncsa.illinois.edu'): # NCSA\n",
    "    repo = \"/repo/dc2\"\n",
    "else:\n",
    "    raise Exception(f\"Unrecognized URL: {URL}\")\n",
    "\n",
    "# Collection should be the same on all sites\n",
    "collection='2.2i/runs/DP0.1'\n",
    "\n",
    "# Define a dictionary with the band, ccd, and visit we wish to view\n",
    "dataId = {'band':'i', 'visit':512055, 'detector':76}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of the butler to get the catalog\n",
    "butler = dafButler.Butler(repo,collections=collection)\n",
    "    \n",
    "# use the dataId and the 'src' to get the source catalog. \n",
    "source_cat = butler.get('src', **dataId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find the path to the file again with the `butler.getURI` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We want to grab the dataset reference for this dataId \n",
    "#  to pass to the gen3 butler.getURI method...\n",
    "# For this dataId, there really should only be one entry;\n",
    "#  so really not necessary to loop...\n",
    "datasetRefs = butler.registry.queryDatasets(datasetType='src',dataId=dataId, collections=collection)\n",
    "for i,ref in enumerate(datasetRefs):\n",
    "    try: uri = butler.getURI(ref)\n",
    "    except: print(\"File not found...\")\n",
    "    if i > 10: break\n",
    "uri = butler.getURI(ref)\n",
    "print(uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtypes = butler.registry.queryDatasetTypes()\n",
    "for i,dt in enumerate(dtypes):\n",
    "    print(dt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few comments are in order on questions you may be having about the butler, and the previous cell. First, for the `gen2` butler, there is no good way to know which `dataId`s exist. That means you have to know ahead of time which `dataId`s it makes sense to use. DM fixed this for the `gen3` butler (e.g., one can use the `butler.registry.queryDatasets()` command to find the `dataId`s for different sets of data). Second, the string `'src'` refers to a very specific data product in the DM philosophy, which is a catalog that contains _the results of different measurement algorithms on detected sources on an individual CCD image_. We will meet some other catalogs later in the tutorial. For now, lets get to know this `src` catalog."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### afw source catalog schemas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#check its schema. Heads up, the schema is pretty big\n",
    "source_cat.getSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These schemas tend to be large if many measurement algorithms used. Several algorithms are a part of the base measurement process, like aperture photometry, SDSS shape and centroid measurements. Other measurements are associated with previous calibration steps, like identifying sources to be candidates for PSF modeling. If deblending was run, several outputs from the process are stored in the table as well. Other measurement processess, like shape measurement processes, are considered extensions. All of these measurements often have many fields and analagous flag fields associated with them. To get a list of names of these high level processes only, we can provide the `topOnly=True` keyword arguement to the `getNames` method we met earlier in the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_cat.getSchema().getNames(topOnly=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, this may be too vague. For example, if you ran KSB and HSM shape measurement, both will be lumped into the 'ext' catagory in the output above, but you may wish to search the schema for one in particular. We can use unix-like pattern matching with the `extract()` method to search the schema. This returns a dictionary where the keys are the schema fields whose names match the pattern you specified, and the values are the fields themselves. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_cat.getSchema().extract('*HSM*Psf*')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Schemas in source catalogs include fields for measurements, and the associated flag fields for those measurements.  To tally how many fields, flag fields, and non-flag fields are contained, we can use several count methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nFields = source_cat.schema.getFieldCount()\n",
    "nFlagFields = source_cat.schema.getFlagFieldCount()\n",
    "nNonFlagFields = source_cat.schema.getNonFlagFieldCount()\n",
    "print('the schema contains {} fields, \\\n",
    "{} flags fields and {} non-flag fields'.format(nFields, nFlagFields, nNonFlagFields ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we are just interested in the field names, we can use the `extract()` method, which supports regex pattern matching. `extract()` returns a python dictionary of key-value pairs, where the keys are the names of the schema fields, and the values are the schema items themselves. We are going to tack on the `keys` method to this dictionary so we just get the ids back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k in source_cat.getSchema().extract('*HSM*Psf*').keys():\n",
    "    print(k)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you already know the id of the field you are interested in, schema's have a `find` method that will return the field in question. For example, it is a safe bet that a schema will contain an id field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_cat.getSchema().find('id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we dumped the entire schema, the very bottom of the schema contained fields  are named 'slot_'. These are called aliases in the schema, and can help you deal with any ambiguity in the table. For example, there are several algorithms used to measure the centroid, and many fileds with 'centroid' in their name as a result. If you want to have quick access to one algorithms measurement result, you can set up a slot alias for it. Lets do a working example on the first record in our table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "slot_centroid = source_cat[0].getCentroid()\n",
    "naive_cent_x, naive_cent_y = (source_cat['base_NaiveCentroid_x'][0], source_cat['base_NaiveCentroid_y'][0])\n",
    "sdss_cent_x, sdss_cent_y = (source_cat['base_SdssCentroid_x'][0], source_cat['base_SdssCentroid_y'][0])\n",
    "\n",
    "print('sloan centroid is {}, {}'.format(sdss_cent_x, sdss_cent_y))\n",
    "print('naive centroid is {}, {}'.format(naive_cent_x, naive_cent_y))\n",
    "print('slot centroid is {}'.format(slot_centroid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# aliasing works with other methods\n",
    "psf_flux_key = source_cat.getPsfFluxSlot().getMeasKey()\n",
    "id_key = source_cat.getIdKey()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As advertised, the slot centroid and SDSS centroid are the same. We also used some syntactic sugar to access the `naive` centroids and `sdss` centroids, which will be familiar to you if you are an astropy tables user. \n",
    "\n",
    "Speaking of astropy tables, you can make an astropy table version of a source catalog:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_cat.asAstropy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this vein, you may find it more convenient to grab columns from the table and work with them as numpy arrays. As we mentioned above, the defacto unit choice of DM is radians. If you would prefer degrees, you can do something like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ra_deg = np.rad2deg(source_cat['coord_ra'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will set aside the schema for this table, and look at the table itself so we can examine its methods.\n",
    "\n",
    "### afw source catalogs\n",
    "\n",
    "Source catalogs support a lot of fast operations for common use cases which we will now discuss. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sorting is supported. by default catalogs are sorted by id\n",
    "source_cat.isSorted(id_key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find aperture flux columns\n",
    "[k for k in source_cat.getSchema().extract('*ApFlux*').keys()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can cut on the catalog.\n",
    "# e.g. Make a boolean array to only keep sources with positive psf flux:\n",
    "psf_mask = source_cat['slot_PsfFlux_instFlux'] > 0\n",
    "psf_mask &= np.isfinite(source_cat['slot_ApFlux_instFlux'])\n",
    "psf_mask &= np.isfinite(source_cat['slot_ApFlux_instFluxErr'])\n",
    "psf_mask &= np.isfinite(source_cat['base_ClassificationExtendedness_value'])\n",
    "pos_flux = source_cat.subset(psf_mask)\n",
    "\n",
    "# You can sort on other keys too:\n",
    "flux_key = pos_flux.getPsfFluxSlot().getMeasKey()\n",
    "pos_flux.sort(flux_key)\n",
    "pos_flux.isSorted(flux_key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can get the children of particular objects.\n",
    "# This is useful if you want to understand how one object was deblended, for example:\n",
    "\n",
    "print(source_cat.getChildren(34363434455795246)) #the argument is the id of the parent object\n",
    "\n",
    "# Note that this will only work if the source catalog is sorted on id or parent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check if catalogs are contiguous in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_cat.isContiguous()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_flux.isContiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some operations are quicker if catalogs are contiguous in memory, like using numpy-like syntax to create masks. Eli Rykoff performed some benchmark tests showing this is the case for a catalog with about half a million enteries. You can find the full details in a Slack thread [here](https://lsstc.slack.com/archives/C2JPL2DGD/p1525799998000344). Additionally, certain operations will not work on non-contiguous tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# uncomment the last line if you are curious and want to prove it wont work\n",
    "# this will not work and give you an error message complaining pos_flux is not contiguous\n",
    "# pos_flux.asAstropy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can always force a table to be contiguous by making a deep copy of it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_flux = pos_flux.copy(deep=True)\n",
    "pos_flux.isContiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next few cells, we show a few methods that can be used to search through tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the between method to get the indices of values within a range:\n",
    "pos_flux.between(1e4,1e5,psf_flux_key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The slice object tells you the (start, stop, stride) for values that fit our query.\n",
    "# You can check to see that the first record outside the slice is above the flux threshold\n",
    "print(pos_flux[1076].get(psf_flux_key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and that the last element in the slice is inside the threshold\n",
    "print(pos_flux[1584].get(psf_flux_key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your turn. confirm the lower limits of the between query\n",
    "# pos_flux[...].getPsfFlux\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#the upper and lower bound methods work similarly\n",
    "pos_flux.upper_bound(1e4, psf_flux_key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_flux.lower_bound(1e5, psf_flux_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Star-Galaxy Separation\n",
    "\n",
    "Now that we have introduced the functionality of the source catalog and its schema, we will do a toy example of star-galaxy separation. This small demo will also flags and fields that users are use, and ultimately make a plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's select sources that were not deblended\n",
    "select_mask = source_cat['deblend_nChild'] == 0\n",
    "select_mask &= source_cat['parent'] == 0\n",
    "\n",
    "# use the extendedness column for a mock star/galaxy seperation\n",
    "# we only want to use columns where this algorithm worked\n",
    "# the flag is set true if there was a failure, so we invert the flag values here\n",
    "select_mask &= ~source_cat['base_ClassificationExtendedness_flag']\n",
    "\n",
    "# we will also use the sloan shapes to measure size\n",
    "select_mask &= ~source_cat['base_SdssShape_flag']\n",
    "\n",
    "# and a simple aperture flux for the brightness\n",
    "select_mask &= ~source_cat['base_CircularApertureFlux_12_0_flag']\n",
    "\n",
    "# only consider sources with positive flux\n",
    "select_mask &= source_cat['base_CircularApertureFlux_12_0_instFlux'] > 0\n",
    "\n",
    "size_bright_cat = source_cat.subset(select_mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we make a crude size magnitude diagram, color coding the data by their 'extendedness value'. The extendedness will be 1 for extended sources-like galaxies-and 0 for point sources-like stars. One hopes the stars will all live on the stellar locus..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(np.log(size_bright_cat['base_CircularApertureFlux_12_0_instFlux']), \n",
    "           size_bright_cat['base_SdssShape_xx'] + size_bright_cat['base_SdssShape_yy'],\n",
    "           c=size_bright_cat['base_ClassificationExtendedness_value'],\n",
    "            cmap='bwr',\n",
    "           s=4)\n",
    "plt.xlabel('log flux')\n",
    "plt.ylabel('size px^2')\n",
    "plt.ylim([0,20]) #zoom in to make the stellar locus clearer\n",
    "plt.xlim([5,15])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our plot shows some star galaxy separation, but also has other interesting features. Some detected sources appear to be smaller than the PSF, some of the point sources have a (crudely) calculated size that occupy the same parameter space as extended sources, and there are a few extremely faint detected point sources. We will leave it to you to delve into this mystery further as a homework assignment since we are primarily focused on understanding tables in this tutorial. By making this plot we exercised some of the methods of the catalog and its schema, to do a minimal analysis example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operations with multiple tables/catalogs\n",
    "\n",
    "In the next section we will show operations which involve two or more catalogs.\n",
    "\n",
    "#### Table concatenation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grab a second catalog using the butler:\n",
    "dataId2 = {'band':'i', 'visit':512055, 'detector':75}\n",
    "source_cat2 = butler.get('src', dataId2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put our catalogs in a list:\n",
    "catalogList = [source_cat, source_cat2]\n",
    "\n",
    "# The following concatenation function is courtesy of Jim Bosch:\n",
    "def concatenate(catalogList):\n",
    "    from functools import reduce\n",
    "    \"\"\"Concatenate multiple catalogs (FITS tables from lsst.afw.table)\"\"\"\n",
    "    schema = catalogList[0].schema\n",
    "    for i, c in enumerate(catalogList[1:]):\n",
    "        #check that the schema in the tables are the same\n",
    "        #we can only cat them if this is true\n",
    "        if c.schema != schema:\n",
    "            raise RuntimeError(\"Schema for catalog %d not consistent\" % (i+1))\n",
    "\n",
    "    # prepare the master catalog\n",
    "    out = afwTable.BaseCatalog(schema)\n",
    "    num = reduce(lambda n, c: n + len(c), catalogList, 0)\n",
    "    # set aside enough space for all the records and their pointers\n",
    "    out.reserve(num)\n",
    "\n",
    "    # stick in all the records from all catalogs into the master catalog\n",
    "    for catalog in catalogList:\n",
    "        for record in catalog:\n",
    "            out.append(out.table.copyRecord(record))\n",
    "\n",
    "    return out\n",
    "\n",
    "cat_source = concatenate(catalogList)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Catalog matching\n",
    "\n",
    "Quick positional matching is supported by the stack, and offers some useful functionality. In the next example, we will match two overlapping observations from different filters together. Getting this data will require us to use a new data repository, so we will have to set up a new butler, then ask for our two tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For the rest of this tutorial, we are going to work with coadd data products\n",
    "tract = 4851\n",
    "patch = 29   #gen3 analog to gen2 patch id '1,4'\n",
    "bandList = [\"i\", \"r\"]\n",
    "filterList = bandList   # using filterList for consistency with cells folllowing below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's grab some forced photometry for the two bands. They will have the same tract and patch, which ensures they will overlap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "objects = []\n",
    "for filter in filterList:\n",
    "    dataId = {'band':filter, 'tract':tract, 'patch':patch}\n",
    "    objects.append(butler.get(\"deepCoadd_forced_src\", dataId=dataId))\n",
    "iSources, rSources = objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need these calib objects for our two bands so that we can calculate magnitudes a little later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#dataId = {'band':filter, 'tract':tract, 'patch':patch}\n",
    "#dcal = butler.get(\"deepCoadd_calexp\", dataId=dataId)\n",
    "#calib = dcal.getPhotoCalib()\n",
    "#print(calib)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calibs = []\n",
    "for filter in filterList:\n",
    "    dataId = {'band':filter, 'tract':tract, 'patch':patch}\n",
    "    dcal = butler.get(\"deepCoadd_calexp\", dataId=dataId)\n",
    "    calib = dcal.getPhotoCalib()\n",
    "    calibs.append(calib)\n",
    "iCalib, rCalib = calibs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some quality control flags to prune down the data to give us stars with a signal to noise ratio of 10 or higher. We will use this to index into our catalogs when we do the matching. Note that because these are forced photometry catalogs, the records in each catalog line up so that they should be refering to the same astrophysical sources. That is why we will be able to use our mask on both iSources and rSources. This is not true in general of afwTables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "noChildren = iSources['deblend_nChild'] == 0\n",
    "isGoodFlux = ~iSources['modelfit_CModel_flag']\n",
    "isStellar = iSources['base_ClassificationExtendedness_value'] < 1.\n",
    "snr = iSources['modelfit_CModel_instFlux']/iSources['modelfit_CModel_instFluxErr'] > 10\n",
    "\n",
    "star_flag = noChildren & isGoodFlux & isStellar & snr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to match catalogs, we must provide a `MatchControl` instance. The `MatchControl` provides configurations for catalog matching. It has three 'switches' in the form of class attributes. They are defined as follows:\n",
    "\n",
    "1. `findOnlyClosest`: True by default. If False, all other sources within a search radius are also matched \n",
    "2. `includeMismatches`: False by default. If False, sources with no match are not reported in the match catalog. If True, sources with no match are included in the match catalog with Null as their match\n",
    "3. `symmetricMatch`: False by default. If False, the match between source a from catalog a with source b from catalog b is reported alone. If True, the symmetric match between source b and a is also reported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get a match control, we will keep the default configuration\n",
    "mc = afwTable.MatchControl()\n",
    "\n",
    "# match our two catalogs, setting the match threshold to be one arcsecond\n",
    "matches = afwTable.matchRaDec(iSources[star_flag], rSources[star_flag],\n",
    "                              lsst.geom.Angle(1,lsst.geom.arcseconds), mc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`afwTable.matchRaDec` returns a list, where each element is an instance of a `Match` class. The `Match` class has three attributes, which gives us information about the matched sources. Let us unpack this a bit before moving on to some analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# how many sources were actually matched?\n",
    "len(matches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets examine the first element in the matches list\n",
    "# we can grab the record corresponding to this source in the i band catalog\n",
    "# using the first attribute\n",
    "matches[0].first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# likewise the second attribute gives us the record from the r band catalog\n",
    "matches[0].second"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#finally the angular seperation is given in radians in the distance attribute\n",
    "matches[0].distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have matched stars in two bands, lets make a color magnitude diagram, displaying how to use matches in a little more depth\n",
    "first we start out simple and demonstrate how to get the magnitude and magnitude error for one record in our matches object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the calib object to get magnitudes\n",
    "# pass in a record and the name of the flux field you are interested in\n",
    "iCalib.instFluxToMagnitude(matches[0].first, 'modelfit_CModel')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now we make some loops to grab all the magnitudes in the iband and rband\n",
    "# remember that the i band catalog is accessed with the first attribute\n",
    "# and the r band catalog is accessed with the second attribute\n",
    "iMag = [iCalib.instFluxToMagnitude(m.first, 'modelfit_CModel').value for m in matches]\n",
    "rMag = [rCalib.instFluxToMagnitude(m.second, 'modelfit_CModel').value for m in matches]\n",
    "\n",
    "plt.scatter(np.array(rMag) - (iMag), iMag)\n",
    "plt.ylim([26,18])\n",
    "plt.xlim([-0.5,3])\n",
    "plt.xlabel('$r-i$')\n",
    "plt.ylabel('$i$')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are only interested in the ids and the angular separation, you can pack the matches into a table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matches_table = afwTable.packMatches(matches)\n",
    "matches_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can unpack the matches too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unpack_matches = afwTable.unpackMatches(matches_table, iSources, rSources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully this gives you some idea of the power of `afwTable`s in matching catalogs together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this tutorial we introduced afw tables. We introduced schemas, how to navigate them, add to them, and how to create tables based off of them. We covered data access to catalogs produced by DM using the data butler. We went over a some typical use cases for source catalogs, like catalog matching, and understanding bread and butter measurement algorithms."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "livereveal": {
   "scroll": true,
   "start_slideshow_at": "selected"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}