DP02_02c_Image_Queries_with_TAP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bd1cf3c8-c8d1-4867-a73b-ce50d38163f8",
   "metadata": {},
   "source": [
    "# Image Queries with the TAP Service\n",
    "\n",
    "<img align=\"left\" src = https://project.lsst.org/sites/default/files/Rubin-O-Logo_0.png width=250 style=\"padding: 10px\" alt=\"Rubin Observatory logo, a graphical representation of turning stars into data.\">\n",
    "<br>\n",
    "Contact author: Melissa Graham <br>\n",
    "Last verified to run: 2024-07-28 <br>\n",
    "LSST Science Piplines version: Weekly 2024_16 <br>\n",
    "Container Size: medium <br>\n",
    "Targeted learning level: intermediate <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5b0dde7-0f4e-4a44-9dd8-bde55206d24b",
   "metadata": {},
   "source": [
    "**Description:** Retrieve and display images using the ObsTAP service."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99ffa893-908a-42c8-9e9f-7bdebac233f1",
   "metadata": {},
   "source": [
    "**Skills:** Query for image data via the TAP service; retrieve image data with PyVO and Datalink; display and save images."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc76389b-7007-42f5-bff4-d19cc7b8d99d",
   "metadata": {},
   "source": [
    "**LSST Data Products:** calexp, deepCoadd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a5e559d-42d6-42ec-b115-47c7a7b52929",
   "metadata": {},
   "source": [
    "**Packages:** lsst.rsp, lsst.afw, pyvo.dal.adhoc, urllib.request"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cb548ca-e04b-4423-a097-c541f9424710",
   "metadata": {},
   "source": [
    "**Credit:** Originally developed by Melissa Graham and the Rubin Community Science team.\n",
    "Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0c7efad-8b13-453e-beaa-0802f837807b",
   "metadata": {},
   "source": [
    "**Get Support:**\n",
    "Find DP0-related documentation and resources at\n",
    "[dp0.lsst.io](https://dp0.lsst.io).\n",
    "Questions are welcome as new topics in the\n",
    "[Support - Data Preview 0 category](https://community.lsst.org/c/support/dp0) \n",
    "of the Rubin Community Forum. \n",
    "Rubin staff will respond to all questions posted there."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e67efc-f77e-4e4e-abf0-6f58bb3aeb43",
   "metadata": {},
   "source": [
    "## 1. Introduction\n",
    "\n",
    "The simulted images for Rubin's Data Preview 0 (DP0) are available via both the\n",
    "[data butler](https://pipelines.lsst.io/modules/lsst.daf.butler/index.html)\n",
    "and the TAP (Table Access Protocol) service.\n",
    "\n",
    "Use of the butler to query and retrieve images\n",
    "is covered in other tutorials, and is generally the recommended\n",
    "way to access DP0 images while working within the Notebook Aspect of the \n",
    "Rubin Science Platform, especially if any image reprocessing is being done.\n",
    "However, image access via the TAP service is also possible.\n",
    "It is how images are accessed via the\n",
    "[Portal](https://rsp.lsst.io/guides/portal/index.html)\n",
    "and [API](https://rsp.lsst.io/guides/api/index.html) Aspects of\n",
    "the Rubin Science Platform.\n",
    "\n",
    "This tutorial demonstrates how to discover the existence of image\n",
    "data with the TAP service (Section 2),\n",
    "how to write ADQL queries to retrieve image metadata (Section 3),\n",
    "and how to use the PyVO's Datalink package for image data retrieval (Section 4).\n",
    "It also demonstrates how to display the full retrieved images with `matplotlib`,\n",
    "`afwDisplay`, and `Firefly`.\n",
    "Techniques for image display and manipulation, and the creation of image cutouts,\n",
    "are covered in [other DP0.2 tutorials](https://dp0-2.lsst.io/tutorials-examples/index.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bfbc629-5d7f-49ca-93e4-f37cc506a4ad",
   "metadata": {},
   "source": [
    "### 1.1. Known issue with the sky region (`s_region`)\n",
    "\n",
    "The sky region (`s_region`) refers to image metadata that contains the region of sky covered by an image.\n",
    "\n",
    "For the Rubin data, the region is provided as a four-corner polygon in\n",
    "International Celestial Reference System\n",
    "([ICRS](https://en.wikipedia.org/wiki/International_Celestial_Reference_System_and_its_realizations))\n",
    "coordinates of right ascension (RA) and declination (Dec), in decimal degrees.\n",
    "This piece of metadata enables the common image query of \"does this image contain this point\".\n",
    "\n",
    "The `s_region` metadata is of type string, and is formatted like:\n",
    "\n",
    "```\n",
    "POLYGON ICRS 61.693243 -36.780375 61.702475 -37.034112 62.015405 -37.026409 62.005102 -36.772698\n",
    "```\n",
    "\n",
    "**Known Issue:** The `s_region` for individual detectors of single-visit images (e.g., raw, `calexp` images)\n",
    "has some padding and is larger than the actual image boundary.\n",
    "\n",
    "Due to this issue, queries that request all images containing a given coordinate will return images for which\n",
    "the coordinate is in the padding zone, and not in the image.\n",
    "\n",
    "For processed visit images (e.g., `calexp`s), the `s_region` parameter is defined at the time when the visit is defined for observation.\n",
    "It uses the camera geometry and the telescope boresight, and has some padding.\n",
    "Think of it as the anticipated outer boundaries that will contain the sky region imaged by that specific detector,\n",
    "and not based on the image WCS or astrometric solution.\n",
    "\n",
    "Although the image metadata will evolve from what is available for DP0.2, and query by WCS boundaries should become\n",
    "possible, users should be prepared for cases where the coordinate is within the `s_region` but is not covered by image pixel data.\n",
    "For example, detector defects, lost amplifiers, etc. would not affect the `s_region`.\n",
    "\n",
    "This issue is demonstrated in Section 4.1."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ecb5055-4d1e-4cae-9a72-5426d8530be6",
   "metadata": {},
   "source": [
    "### 1.2. DAL Service Errors\n",
    "\n",
    "This tutorial notebook runs a lot of different TAP queries.\n",
    "\n",
    "Users might encounter the following error.\n",
    "\n",
    "```\n",
    "DALServiceError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))\n",
    "```\n",
    "\n",
    "Typically this is fixed by rerunning the cell with the TAP service fetch statement.\n",
    "If that doesn't fix it, try restarting the kernel and clearing the notebook, and re-executing the cells.\n",
    "(In the menu, click on \"Kernel\" and then \"Restart Kernal and Clear Outputs of All Cells...\")."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "678c2593-c625-4129-8168-8ec111e81910",
   "metadata": {},
   "source": [
    "### 1.3. Import packages\n",
    "\n",
    "Import packages a few standard python packages,\n",
    "[lsst packages](https://pipelines.lsst.io/) for image display and data access,\n",
    "[astropy packages](https://www.astropy.org/) for working with images,\n",
    "and a module from the [pvyo.dal package](https://pyvo.readthedocs.io/en/latest/dal/index.html)\n",
    "for data access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e29ed16-5f40-4118-8c60-74c1afae8e7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from urllib.request import urlretrieve\n",
    "\n",
    "import lsst.afw.display as afwDisplay\n",
    "import lsst.geom as geom\n",
    "from lsst.rsp import get_tap_service\n",
    "from lsst.afw.image import ExposureF\n",
    "\n",
    "import astropy.visualization as vis\n",
    "from astropy.io import fits\n",
    "from astropy.wcs import WCS\n",
    "from astropy.coordinates import SkyCoord\n",
    "from astropy import units as u\n",
    "\n",
    "from pyvo.dal.adhoc import DatalinkResults"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cfee068-f967-4e45-83a2-2e085297ece8",
   "metadata": {},
   "source": [
    "### 1.4. Define functions and paramters\n",
    "\n",
    "Instantiate the RSP's TAP service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d818a521-b241-45a8-8ca6-0addf8b38051",
   "metadata": {},
   "outputs": [],
   "source": [
    "service = get_tap_service(\"tap\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c58604ec-879e-4b26-ba25-97cbeceaf71c",
   "metadata": {},
   "source": [
    "Define `auth_session` in order to authorize the TAP service to\n",
    "access LSST images via Datalink in Section 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad507d6f-537c-472c-864e-17fe2ea9acb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "auth_session = service._session"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ea0ac5d-63e2-4715-82a5-913163b0c693",
   "metadata": {},
   "source": [
    "Define a point near the center of the Data Preview 0 area with\n",
    "coordintes Right Ascension (RA) = 62 degrees\n",
    "and Declination (Dec) = -37 degrees.\n",
    "Use several different formats for the different use cases\n",
    "throughout this tutorial\n",
    "\n",
    " * `target_ra`, `target_dec` : floats, to be used in calculations\n",
    " * `target_str_ra`, `target_str_dec` : strings, to be inserted into query statements\n",
    " * `targetPoint` : a `SpherePoint` object from the LSST Science Pipelines' `geom` package\n",
    " * `targetCoord` : an astropy `SkyCoord` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62d01de1-7103-4a24-bf0e-1159d1af4ae0",
   "metadata": {},
   "outputs": [],
   "source": [
    "target_ra = 62.0\n",
    "target_dec = -37.0\n",
    "target_str_ra = '62.0'\n",
    "target_str_dec = '-37.0'\n",
    "targetPoint = geom.SpherePoint(target_ra*geom.degrees,\n",
    "                               target_dec*geom.degrees)\n",
    "targetCoord = SkyCoord(ra=target_ra*u.degree,\n",
    "                       dec=target_dec*u.degree,\n",
    "                       frame='icrs')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f335a9-0d7f-4c51-b513-cf425aa0a8ca",
   "metadata": {},
   "source": [
    "Define a function `sregion_to_vertices` to be used in Section 3.5. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "935f7209-8b38-4b9b-abd8-e69c0102a706",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sregion_to_vertices(sregion: str):\n",
    "    \"\"\"Convert the s_region from the ObsCore table into two\n",
    "    arrays containing the x and y vertices, in order to plot\n",
    "    boxes using matplotlib.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    str_polygon: `str`\n",
    "        String formatted with 10 space-separated items, e.g.,\n",
    "        the \"s_region\" column from the ObsCore table which has\n",
    "        the words \"POLYGON ICRS\" followed by 8 numbers:\n",
    "        \"POLYGON ICRS # # # # # # # #\".\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    x_vertices: `np.ndarray`\n",
    "        The array of x-vertices.\n",
    "    y_vertices: `np.ndarray`\n",
    "        The array of y-vertices.\n",
    "    \"\"\"\n",
    "    temp = sregion.split(' ')\n",
    "    xvertices = []\n",
    "    yvertices = []\n",
    "    ix = 2\n",
    "    iy = 3\n",
    "    for c in range(4):\n",
    "        xvertices.append(float(temp[ix]))\n",
    "        yvertices.append(float(temp[iy]))\n",
    "        ix += 2\n",
    "        iy += 2\n",
    "    xvertices.append(xvertices[0])\n",
    "    yvertices.append(yvertices[0])\n",
    "    return xvertices, yvertices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca205fc3-0aba-4f1b-a9cf-6aaf8737c106",
   "metadata": {},
   "source": [
    "## 2. Explore the TAP schema for images\n",
    "\n",
    "Retrieve all available schemas and display them as a [pandas](https://pandas.pydata.org/) table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb941fa2-994e-4f0c-9384-fb9f83246e64",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT * FROM tap_schema.schemas\"\n",
    "results = service.search(query)\n",
    "results.to_table().to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4abf087-c93d-4e8e-bdd3-21ce8bd0c23c",
   "metadata": {},
   "source": [
    "The fact that catalogs are available with `schema_name` = `dp02_dc2_catalogs` is pretty clear in the above table.\n",
    "\n",
    "Although it is not clear, DP0 images are stored in `ObsTAP`, with `schema_name` = `ivoa`.\n",
    "This will be clarified for future data releases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8924fd50-8d45-4b43-8045-79c69f9e1a1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "del query, results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8463597f-8c70-479e-8187-9cbd046db21c",
   "metadata": {},
   "source": [
    "### 2.1. The IVOA schema\n",
    "\n",
    "IVOA stands for [International Virtual Observatory Alliance](https://ivoa.net/), and they work towards defining standards\n",
    "for astronomical data storage that enable wide data sharing.\n",
    "The LSST Science Pipelines use IVOA standards.\n",
    "\n",
    "Retrieve all of the tables available in the `ivoa` schema and display them as a pandas table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "951f4ae4-c0d4-4064-b10a-e2f1f8add499",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT * \" \\\n",
    "        \"FROM tap_schema.tables \" \\\n",
    "        \"WHERE tap_schema.tables.schema_name = 'ivoa'\"\n",
    "results = service.search(query)\n",
    "results.to_table().to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35361f18-579a-40cc-8caf-af28201869a3",
   "metadata": {},
   "source": [
    "There is only one `ivoa` table available for DP0: `ObsCore`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5738f062-0e63-43a4-a43e-e8032b4e2e59",
   "metadata": {},
   "outputs": [],
   "source": [
    "del query, results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16d98269-9dba-468f-84dc-8c0548926728",
   "metadata": {},
   "source": [
    "### 2.2. The ObsCore table\n",
    "\n",
    "The LSST `ObsCore` table is essentially a view into the images\n",
    "stored in the LSST's [data butler](https://pipelines.lsst.io/modules/lsst.daf.butler/index.html) registry (e.g., [DMTN-236](https://dmtn-236.lsst.io/)).\n",
    "\n",
    "Retrieve the names of the columns and their data types, descriptions and units,\n",
    "and display these properties as a pandas table.\n",
    "\n",
    "The output of the following cell is a list of the image metadata \n",
    "that is available in the `ObsCore` table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34f5814a-fdf2-4694-b7a0-def0563b4f1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT column_name, datatype, description, unit \" \\\n",
    "        \"FROM tap_schema.columns \" \\\n",
    "        \"WHERE table_name = 'ivoa.ObsCore'\"\n",
    "results = service.search(query)\n",
    "results.to_table().to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53cd2fcf-60a9-40d9-95f2-f6b6193f7971",
   "metadata": {},
   "outputs": [],
   "source": [
    "del query, results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd1edafd-37ae-45ad-b2d4-4d44c8bf5821",
   "metadata": {},
   "source": [
    "## 3. Image metadata\n",
    "\n",
    "Explore the metadata for all images that overlap the target point\n",
    "defined in Section 1.4.\n",
    "\n",
    "Now that the queries are for data (not just schemas),\n",
    "and might take longer, use asynchronous queries (i.e., use the job service as shown below).\n",
    "\n",
    "The following query selects all columns from the `ObsCore` table\n",
    "for all images that contain the\n",
    "[ICRS](https://en.wikipedia.org/wiki/International_Celestial_Reference_System_and_its_realizations)\n",
    "coordinate 62, -37 within the `s_region`.\n",
    "(In other words, where the statment that the `s_region` contains \n",
    "the point is `True` or `= 1`.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fe6c68f-f7bd-42b4-8f71-1a9bddbb0419",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT * FROM ivoa.ObsCore \"\\\n",
    "        \"WHERE CONTAINS(POINT('ICRS', \" + target_str_ra + \\\n",
    "        \", \" + target_str_dec + \"), s_region) = 1\"\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33fe174c-9999-413d-8bdb-b99bc9d810e5",
   "metadata": {},
   "source": [
    "Define the service job using the query, then run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "475171e9-d2e7-44dc-9b41-22fb976b3321",
   "metadata": {},
   "outputs": [],
   "source": [
    "job = service.submit_job(query)\n",
    "job.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f606a312-6501-49ec-9182-8ceaab212706",
   "metadata": {},
   "source": [
    "Wait until the job is completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fb45821-3351-4785-a5a7-cad3a3e1cf21",
   "metadata": {},
   "outputs": [],
   "source": [
    "job.wait(phases=['COMPLETED', 'ERROR'])\n",
    "print('Job phase is', job.phase)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73dfdaf0-03b8-4a17-aac8-97e0697060cc",
   "metadata": {},
   "source": [
    "If the job phase is \"COMPLETED\", move on to the cell below.\n",
    "\n",
    "If the job phase is is \"ERROR\", uncomment the\n",
    "following cell and execute it to see more information about the error.\n",
    "The most common errors are ADQL syntax errors which can be fixed by\n",
    "editing the query and re-submitting the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59826a09-01b7-4530-9361-adcff64d767c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# job.raise_if_error()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63eb91ee-056c-4ae5-8b23-dd54135ccd51",
   "metadata": {},
   "source": [
    "When the job phase is \"COMPLETED\", fetch (retrieve) the results,\n",
    "and print the number of rows returned.\n",
    "This is the number of images found that overlap the specified point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a338825c-2240-48da-949e-9f988740de22",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = job.fetch_result().to_table().to_pandas()\n",
    "print(len(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f67a459-d709-4eca-b3b0-e8321309004d",
   "metadata": {},
   "source": [
    "Option to view the large results table.\n",
    "Notice that the table is truncated both in rows and columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4504ebcd-7dff-4e37-ae88-5ed0ed6f0261",
   "metadata": {},
   "outputs": [],
   "source": [
    "# results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65117056-0359-405b-bdc0-d1d7cac95e0f",
   "metadata": {},
   "source": [
    "### 3.1. Data product types\n",
    "\n",
    "Print the unique values of the column `dataproduct_type`\n",
    "(and the number of rows with that value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4360da7-3cb7-467b-b6a6-31f7831477a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "column = 'dataproduct_type'\n",
    "print('Unique values of: ', column)\n",
    "values, counts = np.unique(results[column], return_counts=True)\n",
    "for value, count in zip(values, counts):\n",
    "    print('%s (%i)' % (value, count))\n",
    "del column, values, counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e67ec1a6-54c7-4452-9226-f638d337a196",
   "metadata": {},
   "source": [
    "Above, notice that all rows have `dataproduct_type` = \"image\".\n",
    "This is as expected, because the `ObsCore` table only holds images\n",
    "\n",
    "### 3.2. Data product subtypes\n",
    "\n",
    "Print the unique values of the column `dataproduct_subtype`\n",
    "(and the number of rows with that value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8368d5d8-692b-441a-b2d2-8ab412be5168",
   "metadata": {},
   "outputs": [],
   "source": [
    "column = 'dataproduct_subtype'\n",
    "print('Unique values of: ', column)\n",
    "values, counts = np.unique(results[column], return_counts=True)\n",
    "for value, count in zip(values, counts):\n",
    "    print('%s (%i)' % (value, count))\n",
    "del column, values, counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dbb44f9-d9c5-4df7-a611-b8b91706cfb2",
   "metadata": {},
   "source": [
    "Above, notice that there are five types of images.\n",
    "\n",
    "The DP0.2 documentation contains more detailed information about the \n",
    "[image subtypes](https://dp0-2.lsst.io/data-products-dp0-2/index.html#images)\n",
    "which are summarized below.\n",
    "\n",
    "1. `calexp` : A reduced and calibrated image (exposure), with the background subtracted. One of the camera's 189 detectors from a processed visit image (PVI).\n",
    "\n",
    "2. `deepCoadd_calexp` : An image created by combining (coadding) `calexp`s to make one deeper image (also has had the background subtracted).\n",
    "\n",
    "3. `goodSeeingCoadd` : An image created by combining the `calexp`s in the top 30% of best image quality (best \"seeing\"). This image is used as the template (reference) for difference image analysis.\n",
    "\n",
    "4. `goodSeeingDiff_differenceExp` : A \"difference image\" created by subtracting the `goodSeeingCoadd` from the `calexp`.\n",
    "\n",
    "5. `raw` : An unreduced, uncalibrated image (exposure)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8892840-72b2-4e94-a145-b1b390b05eca",
   "metadata": {},
   "source": [
    "**Why are there 510 `calexp` but only 12 `deepCoadd` images that overlap coordinate 62, -37?**\n",
    "\n",
    "As the LSST makes many repeated visits to the same area of sky, there will be\n",
    "many individual exposures (`raw`, `calexp`, and difference images) that overlap\n",
    "a given coordinate.\n",
    "In this case, 510 visits covered these coordinates.\n",
    "\n",
    "The LSST Science Pipelines defines a sky map that divides the survey area into\n",
    "\"**tracts**\" (large polygons) and \"**patches**\" (smaller squares about the size\n",
    "of one detector).\n",
    "Coadded images are stored as individual patches, and they all slightly\n",
    "overlap at the edges and corners.\n",
    "Any given point within the survey region will be contained by at least 1\n",
    "patch, and thus by 6 `deepCoadd` images (one per filter, *ugrizy*).\n",
    "\n",
    "Thus, the fact that there are 12 overlapping `deepCoadd` images means\n",
    "that these coordinates are at the edge of a patch, overlapping with \n",
    "another patch."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97341af8-bebd-4d63-bb43-0c2f67c1a95b",
   "metadata": {},
   "source": [
    "### 3.3. Calibration level\n",
    "\n",
    "Print the unique values of the column `calib_level`\n",
    "(and the number of rows with that value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9595a12e-79ae-49c7-b7c4-817a3f619c86",
   "metadata": {},
   "outputs": [],
   "source": [
    "column = 'calib_level'\n",
    "print('Unique values of: ', column)\n",
    "values, counts = np.unique(results[column], return_counts=True)\n",
    "for value, count in zip(values, counts):\n",
    "    print('%s (%i)' % (value, count))\n",
    "del column, values, counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ec1ea88-aeb7-4801-b39f-568afd9b3078",
   "metadata": {},
   "source": [
    "Above, notice that there are three calibration levels.\n",
    "\n",
    "The term `calib_level` is a bit of a misnomer, as it represents\n",
    "the processing done to an image more than any calibrations applied.\n",
    "The `calexp`, `deepCoadd`, and difference images have all\n",
    "been photometrically and astrometrically calibrated.\n",
    "\n",
    "The `calib_level` numbers correspond to:\n",
    "1. Raw images (510).\n",
    "2. Calibrated exposures (510).\n",
    "3. Coadded images and difference images (510 + 12 + 12).\n",
    "\n",
    "This becomes evident by printing the unqiue `calib_level` values\n",
    "for each of the `dataproduct_subtype` values in turn, as done\n",
    "in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5715bfb1-8fee-408c-baa8-604c7e2bc7bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "values, counts = np.unique(results['dataproduct_subtype'],\n",
    "                           return_counts=True)\n",
    "for value, count in zip(values, counts):\n",
    "    print('%s (%i)' % (value, count))\n",
    "    tx = np.where(results['dataproduct_subtype'] == value)[0]\n",
    "    values2, counts2 = np.unique(results['calib_level'][tx],\n",
    "                                 return_counts=True)\n",
    "    del tx\n",
    "    for value2, count2 in zip(values2, counts2):\n",
    "        print('  image type %s (%i)' % (value2, count2))\n",
    "    del values2, counts2\n",
    "    print(' ')\n",
    "del values, counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "945bf317-7a1b-49ae-9221-5fb038c67da3",
   "metadata": {},
   "source": [
    "### 3.4. Band and filter\n",
    "\n",
    "For `calexp`s, print the unique values of `lsst_band` (and the number of rows with that value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0364435e-eff1-4e2d-8cc7-3c7dd222b4ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "tx = np.where(results['dataproduct_subtype'] == 'lsst.calexp')[0]\n",
    "values, counts = np.unique(results['lsst_band'][tx], return_counts=True)\n",
    "for value, count in zip(values, counts):\n",
    "    print('lsst_band %s (%i)' % (value, count))\n",
    "del tx, values, counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "990c38db-4e57-43af-9537-85355f2b6046",
   "metadata": {},
   "source": [
    "For `calexp`s, print the unique values of `lsst_filter` (and the number of rows with that value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cb0eda9-c95e-4be7-affd-4374af9c55b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "tx = np.where(results['dataproduct_subtype'] == 'lsst.calexp')[0]\n",
    "values, counts = np.unique(results['lsst_filter'][tx], return_counts=True)\n",
    "for value, count in zip(values, counts):\n",
    "    print('lsst_filter %s (%i)' % (value, count))\n",
    "del tx, values, counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bce80da-d5ec-44e6-a8ca-0f052400aed4",
   "metadata": {},
   "source": [
    "Above, see that band and filter have slightly different formats but are,\n",
    "essentially, the six LSST filters: *ugrizy*.\n",
    "\n",
    "Sort by the number of visits per filter and print the fraction of visits in that filter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7789e009-ec04-4208-9384-0d37e76bcdd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "tx = np.where(results['dataproduct_subtype'] == 'lsst.calexp')[0]\n",
    "values, counts = np.unique(results['lsst_filter'][tx], return_counts=True)\n",
    "ix = np.argsort(counts)\n",
    "for value, count in zip(values[ix], counts[ix]):\n",
    "    print('lsst_filter %s (%4.2f)' % (value, float(count)/float(len(tx))))\n",
    "del tx, values, counts, ix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74e081f2-0b29-4445-a994-aa87e987a371",
   "metadata": {},
   "source": [
    "Above, the distribution of visits per filter clearly favors the *r* and *i* bands.\n",
    "The survey strategy used for DP0, and thus this exactly filter distribution,\n",
    "is not representative of the final strategy which will be used when operations start.\n",
    "However, the general trend of having more visits in the *r* and *i* bands, and\n",
    "spending the least amount of time in *u*-band, will persist.\n",
    "\n",
    "For `deepCoadd`s, print the unique values of `lsst_band` (and the number of rows with that value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1c35f03-3889-4e1d-a16e-c6a722aef1de",
   "metadata": {},
   "outputs": [],
   "source": [
    "tx = np.where(results['dataproduct_subtype'] == 'lsst.deepCoadd_calexp')[0]\n",
    "values, counts = np.unique(results['lsst_band'][tx], return_counts=True)\n",
    "for value, count in zip(values, counts):\n",
    "    print('lsst_band %s (%i)' % (value, count))\n",
    "del tx, values, counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13364a25-1aa1-4b82-bc57-ba1a7a97d864",
   "metadata": {},
   "source": [
    "The above output of 2 `deepCoadd` per filter overlapping the chosen coordinates\n",
    "is as expected, as described in Section 3.2."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e2b8ae7-c28f-4ae8-aa34-b0c04b0eec41",
   "metadata": {},
   "source": [
    "### 3.5. Visit identifiers\n",
    "\n",
    "The visit identifiers apply to single-visit images only, such as `raw`, `calexp`, and difference images.\n",
    "\n",
    "A \"visit\" is one observation with the camera, at a given time and sky pointing, in a given filter.\n",
    "\n",
    "A \"calexp\" is the calibrated exposure from one of the LSST Science Camera's 189 detectors, for one visit.\n",
    "\n",
    "First, print the number of `calexp`s again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d4bbcb6-111b-4dd3-a214-b9b393add260",
   "metadata": {},
   "outputs": [],
   "source": [
    "tx = np.where(results['dataproduct_subtype'] == 'lsst.calexp')[0]\n",
    "print('number of calexps: ', len(tx))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "930eb49b-c7ad-4c71-b913-fd2d0ed80d6f",
   "metadata": {},
   "source": [
    "For one `calexp`, print the `lsst_visit`, `lsst_detector`, and `lsst_ccdvisitid`.\n",
    "\n",
    "Notice that the `ccdvisitid` is a combination of the visit id and the detector id.\n",
    "The LSST Science Camera has 189 detectors.\n",
    "The `ccdvisitid` uniquely identifies a `calexp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eeebfb2f-0f03-4491-9489-c969cb2427c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 0\n",
    "print(results['lsst_visit'][tx[x]], results['lsst_detector'][tx[x]], results['lsst_ccdvisitid'][tx[x]])\n",
    "del x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d485e0a6-0c90-4ab7-afed-3a5961f4cae7",
   "metadata": {},
   "source": [
    "Print the number of unique visits, and then number of unique `ccdvisitid`s. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d7709ba-901c-437f-a1b3-821a9618d38b",
   "metadata": {},
   "outputs": [],
   "source": [
    "values, counts = np.unique(results['lsst_visit'][tx], return_counts=True)\n",
    "print('number of unique visits: ', len(values))\n",
    "del values, counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bdf18b4-6965-41f5-8f25-1f063fedc50f",
   "metadata": {},
   "outputs": [],
   "source": [
    "values, counts = np.unique(results['lsst_ccdvisitid'][tx], return_counts=True)\n",
    "print('number of unique ccdvisitid: ', len(values))\n",
    "del values, counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e05580b6-ab17-4db9-af2c-acf2d4567c08",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-11T17:31:19.119547Z",
     "iopub.status.busy": "2024-07-11T17:31:19.118761Z",
     "iopub.status.idle": "2024-07-11T17:31:19.129430Z",
     "shell.execute_reply": "2024-07-11T17:31:19.128730Z",
     "shell.execute_reply.started": "2024-07-11T17:31:19.119506Z"
    }
   },
   "source": [
    "As the `ccdvisitid` uniquely identifies a given `calexp`, the number of unique values of `ccdvisitid`\n",
    "matches the number of `calexp`s, as expected.\n",
    "\n",
    "However, the number of unique visits returned is less.\n",
    "This means that some of the `calexp`s returned\n",
    "by the query were obtained in the same visit.\n",
    "It is not possible for a given point on the sky\n",
    "to be contained by more than one detector from the same visit.\n",
    "\n",
    "The underlying issue here is the same one as described in Section 1.1.\n",
    "For visit images (e.g., `calexp`s), the `s_region` has padding around the\n",
    "edges, and it overlaps with neighboring detectors.\n",
    "Instances where the target coordinates are at the edge of a detector, and\n",
    "within the padding of the neighboring detector's `s_region`,\n",
    "both `calexp`s will be returned by the query in Section 4.1.1.\n",
    "\n",
    "This issue is discussed further and visualized in Section 4.1.6, and\n",
    "might make more sense after working through that section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9eea371-eee2-43bd-b26d-fb65d241312d",
   "metadata": {},
   "outputs": [],
   "source": [
    "del tx"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "485e08a2-f0b6-4f15-b804-754c0ac6383f",
   "metadata": {},
   "source": [
    "### 3.6. Patch and tract\n",
    "\n",
    "As described in Section 3.2,\n",
    "the LSST Science Pipelines defines a sky map that divides the survey area into\n",
    "\"tracts\" (large polygons) and \"patches\" (smaller squares about the size\n",
    "of one detector).\n",
    "Tracts are identified by a unique 4-digit integer (e.g., 3831).\n",
    "Patches are identified by a 2-digit number (e.g., 02), but it is not unique.\n",
    "All tracts would have a patch 01, 02, 03, and so on.\n",
    "\n",
    "The concept of patch and tract applies only to the stacked images,\n",
    "which are tesselated (joined) to make wide-area `deepCoadds`.\n",
    "\n",
    "In order to have a string that uniquely identifies a patch,\n",
    "create a new column called `tract_and_patch` that has a format like `3831_02`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "932e7554-5258-4d82-92ef-605e8f08bf9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "temp = []\n",
    "for i in range(len(results)):\n",
    "    temp.append(str(results['lsst_tract'].loc[i]) + '_'\n",
    "                + str(results['lsst_patch'].loc[i]))\n",
    "results['tract_and_patch'] = np.asarray(temp, dtype='str')\n",
    "del temp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eb45a4b-ae3b-4ee8-909a-2605a62a56d9",
   "metadata": {},
   "source": [
    "For `deepCoadd`s, print the unique values of `tract_and_patch` (and the number of rows with that value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8492d88-0ae0-4f7b-a5db-f25f26dbc819",
   "metadata": {},
   "outputs": [],
   "source": [
    "tx = np.where(results['dataproduct_subtype'] == 'lsst.deepCoadd_calexp')[0]\n",
    "values, counts = np.unique(results['tract_and_patch'][tx], return_counts=True)\n",
    "for value, count in zip(values, counts):\n",
    "    print('tract and patch %s (%s)' % (value, count))\n",
    "del tx, values, counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bcbd366-37d1-47ae-aaa9-c628eff0d819",
   "metadata": {},
   "source": [
    "The above output of 2 unique patches overlapping the chosen coordinates,\n",
    "each with 6 images (one per filter),\n",
    "is as expected as described in Section 3.2."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c85fa2a5-970e-47af-ac61-3bc42726c4bb",
   "metadata": {},
   "source": [
    "### 3.7. Center and region\n",
    "\n",
    "In the `ObsCore` table, the image centers are held in the `s_ra` and `s_dec` columns.\n",
    "The region covered by the image is held in the `s_region` column,\n",
    "which is a string that contains the corners of the image.\n",
    "\n",
    "**See Section 1.1 for a known issue with `s_region`**.\n",
    "\n",
    "For the first image of subtype `calexp`, print the central coordinates and their type,\n",
    "and the coordinates of the corners from `s_region`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42a68abd-28e5-4b41-adb2-de08456440d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "tx = np.where(results['dataproduct_subtype'] == 'lsst.calexp')[0]\n",
    "print(results['s_ra'][tx[0]], results['s_dec'][tx[0]])\n",
    "print(type(results['s_ra'][tx[0]]), type(results['s_dec'][tx[0]]))\n",
    "print(results['s_region'][tx[0]])\n",
    "del tx"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa217a75-f76b-4b25-a9fa-7ad66bdf0895",
   "metadata": {},
   "source": [
    "Recall that in Section 1.4, a function was defined called `sregion_to_vertices`, \n",
    "which converts the `s_region` into x-axis vertices and y-axis vertices,\n",
    "to enable plotting image boundaries with `matplotlib`.\n",
    "\n",
    "Create a plot that draws the `s_region` corners of the *u*-band `calexp`s\n",
    "and the `deepCoadd`s which overlap coordinates 62, -37."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "596dabdf-e01e-435b-8dbc-74ee43363c82",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(4, 4))\n",
    "plt.plot(target_ra, target_dec, 'o', ms=5, mew=0, color='black')\n",
    "\n",
    "tx = np.where((results['dataproduct_subtype'] == 'lsst.calexp')\n",
    "              & (results['lsst_band'] == 'u'))[0]\n",
    "for x in tx:\n",
    "    xvals, yvals = sregion_to_vertices(results['s_region'][x])\n",
    "    plt.plot(xvals, yvals, lw=1, alpha=0.2, color='darkviolet')\n",
    "    del xvals, yvals\n",
    "del tx\n",
    "\n",
    "tx = np.where((results['dataproduct_subtype'] == 'lsst.deepCoadd_calexp')\n",
    "              & (results['lsst_band'] == 'u'))[0]\n",
    "for x in tx:\n",
    "    xvals, yvals = sregion_to_vertices(results['s_region'][x])\n",
    "    plt.plot(xvals, yvals, lw=1.5, alpha=0.4, color='black')\n",
    "    center_ra = results['s_ra'][x]\n",
    "    center_dec = results['s_dec'][x]\n",
    "    plt.text(center_ra+0.07, center_dec, results['tract_and_patch'][x])\n",
    "    del xvals, yvals, center_ra, center_dec\n",
    "del tx\n",
    "\n",
    "plt.xlabel('Right Ascension [degrees]')\n",
    "plt.gca().invert_xaxis()\n",
    "plt.ylabel('Declination [degrees]')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b117e57-034a-453b-b84f-deed4a37af3b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-29T17:51:02.945073Z",
     "iopub.status.busy": "2024-06-29T17:51:02.944614Z",
     "iopub.status.idle": "2024-06-29T17:51:02.947815Z",
     "shell.execute_reply": "2024-06-29T17:51:02.947453Z",
     "shell.execute_reply.started": "2024-06-29T17:51:02.945056Z"
    }
   },
   "source": [
    "Figure 1: Above, the *u*-band `calexp` images's sky regions (`s_region`; thin purple lines) and the\n",
    "`deepCoadd` sky regions (thicker black lines, labeled with `tract_and_patch` at their\n",
    "center) that overlap coordinate 62, -37 (black dot).\n",
    "This plot uses the convention north-up, east-left (i.e., RA increases to the left).\n",
    "\n",
    "The `deepCoadd` images for all filters use the same sky map - in other words,\n",
    "the boundaries of a tract and patch are independent of filter.\n",
    "Although the `s_region` for the *u*-band `deepCoadd` was used to draw the boundaries\n",
    "in the plot above, they would have been the same for any chosen filter.\n",
    "\n",
    "Clean up by deleting the query, job, and results used above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdf6d300-35cb-418d-81df-180fe86dcabe",
   "metadata": {},
   "outputs": [],
   "source": [
    "del query, job, results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fcfb586-daed-4d49-acf3-e9dc85281c77",
   "metadata": {},
   "source": [
    "## 4. Image pixel data\n",
    "\n",
    "Retrive image data (headers and pixel data), for a `calexp` (Section 4.1)\n",
    "and a `deepCoadd` (Section 4.2) and display them with a variety of\n",
    "image display tools.\n",
    "\n",
    "No demonstration is provided here for differences images because accessing\n",
    "them is similar to accessing a `calexp`; this is left as an exercise for\n",
    "the user in Section 5.\n",
    "\n",
    "No demonstration is provided here for raw images, as they have a \n",
    "multi-extension format which is more challenging, and covered in \n",
    "a separate tutorial.\n",
    "\n",
    "### 4.1. Calexp\n",
    "\n",
    "#### 4.1.1. TAP query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51cc656b-9ee4-48d1-9553-ef80574fb0ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT * FROM ivoa.ObsCore \"\\\n",
    "        \"WHERE CONTAINS(POINT('ICRS', \" + \\\n",
    "        target_str_ra + \", \" + target_str_dec + \"), s_region) = 1 \"\\\n",
    "        \"AND lsst_band = 'i' \"\\\n",
    "        \"AND calib_level = 2\"\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5737ff89-146e-4cc6-a3a0-148d31518366",
   "metadata": {},
   "outputs": [],
   "source": [
    "job = service.submit_job(query)\n",
    "job.run()\n",
    "job.wait(phases=['COMPLETED', 'ERROR'])\n",
    "print('Job phase is', job.phase)\n",
    "results = job.fetch_result()\n",
    "print(len(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ac59807-4a2c-4c4d-a83e-bfa5bdbfce3f",
   "metadata": {},
   "source": [
    "Option to view results as an `astropy` table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d8df8af-58bc-4037-b84e-8906f6d61777",
   "metadata": {},
   "outputs": [],
   "source": [
    "# results.to_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df946699-4895-48d4-9462-df2db4b94b64",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-29T19:22:47.541786Z",
     "iopub.status.busy": "2024-06-29T19:22:47.541242Z",
     "iopub.status.idle": "2024-06-29T19:22:47.544134Z",
     "shell.execute_reply": "2024-06-29T19:22:47.543751Z",
     "shell.execute_reply.started": "2024-06-29T19:22:47.541768Z"
    }
   },
   "source": [
    "For this demonstration only one of the 128 images is needed,\n",
    "so extract the first row into `results_0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9db9a79c-81d1-46ea-b1e2-e28ee348da8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_0 = results[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "545f9ef0-ad2c-4b58-8773-2d1ce26e04a1",
   "metadata": {},
   "source": [
    "Print selected metadata:\n",
    " * dataproduct subtype\n",
    " * CCD visit identifier (`lsst_ccdvisitid`)\n",
    " * visit identifier and detector number\n",
    " * central RA and Dec\n",
    " * band (filter)\n",
    " * exposure start time `t_min` (a modified julian date, MJD)\n",
    " * `s_region` (corner coordinates)\n",
    " * a unique `obs_id` in the TAP service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "054c2397-130c-4db4-a1d8-17345e3b8e8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(results_0['dataproduct_subtype'])\n",
    "print(results_0['lsst_ccdvisitid'])\n",
    "print(results_0['lsst_visit'], results_0['lsst_detector'])\n",
    "print('%7.4f %8.4f' % (results_0['s_ra'], results_0['s_dec']))\n",
    "print(results_0['lsst_band'])\n",
    "print(results_0['t_min'])\n",
    "print(results_0['s_region'])\n",
    "print(results_0['obs_id'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcb96c21-e96d-4864-b5b0-71918147cd59",
   "metadata": {},
   "source": [
    "#### 4.1.2. Access via Datalink URL\n",
    "\n",
    "Data is accessible via an `access_url` and the format of the\n",
    "data is stored in `access_format`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d351a02-afc4-4643-beaf-5463cf38fe2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(results_0['access_url'])\n",
    "print(results_0['access_format'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ab6a4fc-5bf4-42af-804f-d79c0a7dc574",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-29T19:25:33.079913Z",
     "iopub.status.busy": "2024-06-29T19:25:33.079695Z",
     "iopub.status.idle": "2024-06-29T19:25:33.082247Z",
     "shell.execute_reply": "2024-06-29T19:25:33.081846Z",
     "shell.execute_reply.started": "2024-06-29T19:25:33.079900Z"
    }
   },
   "source": [
    "Above, the format indicates that the image is stored as a `votable` which is accessible via Datalink."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dba817b7-b302-4f1f-baa1-22bbaf22a558",
   "metadata": {},
   "source": [
    "Obtain the table `dl_results` using Datalink and the `auth_session` defined in Section 1.4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "561accb5-628f-4431-86fc-c4f31eb3beac",
   "metadata": {},
   "outputs": [],
   "source": [
    "dl_results = DatalinkResults.from_result_url(results_0['access_url'],\n",
    "                                             session=auth_session)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c787477-da13-4340-a241-bf97a455dad6",
   "metadata": {},
   "source": [
    "Option to view `dl_results` as an astropy table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "957b6c1d-6d4a-4998-80f4-57658f5b7931",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dl_results.to_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "690737e8-087a-4759-b0b7-1a937914835d",
   "metadata": {},
   "source": [
    "The `dl_results` table holds a *different* `access_url` that\n",
    "provides direct access to the image - define it as `image_url`\n",
    "and print it to see how it is different from the `access_url` above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36023847-eaf0-48f6-ad79-383ad9e13add",
   "metadata": {},
   "outputs": [],
   "source": [
    "image_url = dl_results['access_url'][0]\n",
    "print(image_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aa9baf6-22aa-439c-8b7b-0cca11fedef0",
   "metadata": {},
   "source": [
    "#### 4.1.3. Get header data with astropy\n",
    "\n",
    "Image headers are another type of metadata, in addition to what was explored in Section 3.\n",
    "\n",
    "Use the `astropy.fits` package to open the image file using its URL\n",
    "and print the `hdulist` to see the available headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50340c1b-2b23-4853-9a1d-c0f6ca062473",
   "metadata": {},
   "outputs": [],
   "source": [
    "hdulist = fits.open(image_url)\n",
    "for hdu in hdulist:\n",
    "    print(hdu.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fc2a7dd-1cf8-428f-ab81-2451d913c836",
   "metadata": {},
   "source": [
    "There are two headers named \"IMAGE\".\n",
    "The first contains information about the observation, such as the \n",
    "telescope, instrument, airmass, date, and exposure time.\n",
    "\n",
    "Option to print the first IMAGE header of observation information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e0df055-466f-4359-a473-74e657c0fed7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# hdulist[0].header"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e2b5c07-d9d8-4297-a04f-8619eb083d32",
   "metadata": {},
   "source": [
    "The second IMAGE header contains information about the exposure,\n",
    "such as the dimensions, pixel scale, world coordinate system (WCS)\n",
    "transformations, and any scaling factors applied to the pixel fluxes.\n",
    "\n",
    "Option to print the second IMAGE header of exposure information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d9a7889-c429-4562-b606-aaf6be5ffa49",
   "metadata": {},
   "outputs": [],
   "source": [
    "# hdulist[1].header"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "865804bd-bd84-48d6-849f-ecf2a6a25e95",
   "metadata": {},
   "source": [
    "Define `img_hdr` as the second IMAGE header with exposure information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9cf454d-0d43-4f65-aac0-c88d64e07d01",
   "metadata": {},
   "outputs": [],
   "source": [
    "img_hdr = hdulist[1].header"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1565c9c8-6242-4b4d-88cf-216b8c8c64bd",
   "metadata": {},
   "source": [
    "#### 4.1.4. Get image WCS from the header\n",
    "\n",
    "Use the astropy `WCS` package to define the WCS based on header data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc823dfb-f86f-4f35-9138-fde73b20f2b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "img_wcs = WCS(img_hdr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ddbfabd-544d-4c8b-8e27-aeef83243a8e",
   "metadata": {},
   "source": [
    "Use the method `footprint_contains`, which requires a `SkyCoord` be passed, to find out if the image\n",
    "contains the target coordinates, or if they were in the `s_region` padding zone (see Section 1.1.)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79e241d4-28cc-4200-a6a7-51b705a78d42",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(img_wcs.footprint_contains(targetCoord))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70ddca3e-dd01-4b4b-bdd3-587ee61f9bb2",
   "metadata": {},
   "source": [
    "Above, the `False` return means that the target coordinates are inside the `s_region` but outside the image boundaries.\n",
    "\n",
    "Continue with this image to investigate further."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c293479-75fa-4b24-9471-c0fdf6de73f7",
   "metadata": {},
   "source": [
    "#### 4.1.5. Get pixel data with astropy\n",
    "\n",
    "Obtain the `image_data` (pixel flux data) via the `image_url`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c68718b0-60bc-4566-be06-e97b174a5a08",
   "metadata": {},
   "outputs": [],
   "source": [
    "img_data = fits.getdata(image_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81622a04-429e-4f7d-a50d-3a22ca2faed0",
   "metadata": {},
   "source": [
    "#### 4.1.6. Display with matplotlib\n",
    "\n",
    "Use `matplotlib.imshow` to display the pixel data, using the WCS projection from the header to appropriately label the x (RA) and y (Dec) axes.\n",
    "\n",
    "Define the greyscale limits (i.e., pixel flux values corresponding\n",
    "to black and white) using the `astropy.visualization` package's\n",
    "[`ZScaleInterval` class](https://docs.astropy.org/en/stable/api/astropy.visualization.ZScaleInterval.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e3b3976-3510-46e0-8f17-028728e84dc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "zscale = vis.ZScaleInterval()\n",
    "zlimits = zscale.get_limits(img_data)\n",
    "print(zlimits)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35fa7e62-2525-4ce7-b536-999a5c96efab",
   "metadata": {},
   "source": [
    "Plot the image pixel data, and project the WCS with grid lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "707ad92e-803b-4701-b2b8-bd3a0ad397df",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, figsize=(5, 5))\n",
    "plt.subplot(projection=img_wcs)\n",
    "plt.imshow(img_data, cmap='gray',\n",
    "           vmin=zlimits[0], vmax=zlimits[1])\n",
    "plt.axis('on')\n",
    "plt.grid(color='grey', ls='solid')\n",
    "ax.set_xticks([])\n",
    "ax.set_yticks([])\n",
    "plt.xlabel('X')\n",
    "plt.ylabel('Y')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6be1fbe2-8c72-44b3-b2ad-d91b8dd06bae",
   "metadata": {},
   "source": [
    "Figure 2: Plotting the pixel data in X and Y, with grid lines of RA and Dec overplotted.\n",
    "However, the default labeling leaves the image's sky orientation too unclear.\n",
    "\n",
    "Create a more informative plot, below.\n",
    "\n",
    "Use the image WCS to convert the target coordinates and the\n",
    "`s_region` image boundaries from sky coordinates to \n",
    "pixel coordinates, and overplot as an orange circle and\n",
    "purple lines, respectively.\n",
    "\n",
    "Draw 90\"-long lines from the image center (from Section 4.1.1)\n",
    "in the north (N; yellow) and east (E; cyan) directions to provide\n",
    "both a sense of image scale and orientation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17a47368-9b31-47ca-8e2b-4d58df788b10",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, figsize=(5, 5))\n",
    "plt.subplot(projection=img_wcs)\n",
    "plt.imshow(img_data, cmap='gray',\n",
    "           vmin=zlimits[0], vmax=zlimits[1])\n",
    "plt.axis('on')\n",
    "plt.grid(color='grey', ls='solid')\n",
    "\n",
    "targetPixels = img_wcs.world_to_pixel(targetCoord)\n",
    "plt.plot(targetPixels[0], targetPixels[1], 'o', ms=10, mew=2,\n",
    "         color='None', mec='darkorange')\n",
    "del targetPixels\n",
    "\n",
    "xvals, yvals = sregion_to_vertices(results_0['s_region'])\n",
    "xpix = []\n",
    "ypix = []\n",
    "for xval, yval in zip(xvals, yvals):\n",
    "    tempCoord = SkyCoord(ra=xval*u.degree, dec=yval*u.degree, frame='icrs')\n",
    "    tempPixels = img_wcs.world_to_pixel(tempCoord)\n",
    "    xpix.append(tempPixels[0])\n",
    "    ypix.append(tempPixels[1])\n",
    "    del tempCoord, tempPixels\n",
    "plt.plot(xpix, ypix, lw=1, color='darkviolet')\n",
    "del xvals, yvals, xpix, ypix\n",
    "\n",
    "cra = results_0['s_ra'] * u.degree\n",
    "cdec = results_0['s_dec'] * u.degree\n",
    "offset = (90. / 3600.) * u.degree\n",
    "cPix = img_wcs.world_to_pixel(SkyCoord(cra, cdec, frame='icrs'))\n",
    "oraPix = img_wcs.world_to_pixel(SkyCoord(cra + offset, cdec, frame='icrs'))\n",
    "odecPix = img_wcs.world_to_pixel(SkyCoord(cra, cdec + offset, frame='icrs'))\n",
    "plt.plot([cPix[0], oraPix[0]], [cPix[1], oraPix[1]],\n",
    "         ls='solid', lw=1, color='cyan')\n",
    "plt.plot([cPix[0], odecPix[0]], [cPix[1], odecPix[1]],\n",
    "         ls='solid', lw=1, color='yellow')\n",
    "plt.text(oraPix[0], oraPix[1], 'E', color='cyan')\n",
    "plt.text(odecPix[0], odecPix[1], 'N', color='yellow')\n",
    "del cra, cdec, offset, cPix, oraPix, odecPix\n",
    "\n",
    "ax.set_xticks([])\n",
    "ax.set_yticks([])\n",
    "plt.xlabel('X')\n",
    "plt.ylabel('Y')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2012f7b1-de72-480c-929a-2428dc1f2b92",
   "metadata": {},
   "source": [
    "Figure 3: The `calexp` displayed in greyscale, with grey grid lines for RA (30s) and Dec (5') overplotted to demonstrate that individual exposures are obtained with camera rotation (as seen in Figure 1). The target coordinates (04h 08m 00s, -37d 00m 00s orange circle) are just off the edge of this `calexp`, yet within the boundaries of the `s_region`. In the center, 90\"-long lines extend in the north (N; yellow) and east (E; cyan) directions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e72d5e4-d65c-401c-877e-ced72d83376d",
   "metadata": {},
   "source": [
    "Clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fe3813c-843f-4f78-9d6e-23157376ec43",
   "metadata": {},
   "outputs": [],
   "source": [
    "del results_0, dl_results, image_url, hdulist\n",
    "del img_hdr, img_wcs, img_data\n",
    "del zscale, zlimits"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54c7b6d1-d508-46de-affb-dc11b2c76c6a",
   "metadata": {},
   "source": [
    "#### 4.1.7. Select a different image to save and display\n",
    "\n",
    "For the first five images in `results`, get the image WCS from the header and use\n",
    "the `footprint_contains` method to check if the target is within the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f06632e9-472f-413a-9c7a-5075c6698116",
   "metadata": {},
   "outputs": [],
   "source": [
    "for r, result in enumerate(results):\n",
    "    if r < 5:\n",
    "        dl_results = DatalinkResults.from_result_url(result['access_url'],\n",
    "                                                     session=auth_session)\n",
    "        image_url = dl_results['access_url'][0]\n",
    "        hdulist = fits.open(image_url)\n",
    "        img_hdr = hdulist[1].header\n",
    "        img_wcs = WCS(img_hdr)\n",
    "        print(r, img_wcs.footprint_contains(targetCoord))\n",
    "        del dl_results, image_url, hdulist, img_hdr, img_wcs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6282384c-243c-450e-b817-6f62d17fd0eb",
   "metadata": {},
   "source": [
    "Above, the second image (with index `r = 1`) does contain the target.\n",
    "\n",
    "Get the `image_url` for the second image in the `results` list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "940ce170-eed7-4109-9537-94f401d4f4c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = 1\n",
    "result = results[r]\n",
    "dl_results = DatalinkResults.from_result_url(result['access_url'],\n",
    "                                             session=auth_session)\n",
    "image_url = dl_results['access_url'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52e02fe0-96e7-43e7-8dd3-4b5922d0ef84",
   "metadata": {},
   "source": [
    "#### 4.1.8. Save image locally\n",
    "\n",
    "Use the `urlretrieve` module to retrieve the image from\n",
    "its URL and save it as a file in the home directory.\n",
    "Print the filename where the image was saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4db65bdc-f1e0-4cbf-99db-dc8cd7d2a438",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = os.path.join(os.getenv(\"HOME\"), 'dp02_02c_image_calexp.fits')\n",
    "urlretrieve(image_url, filename)\n",
    "print(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23d80b61-27cc-405b-8168-76fc3b2b73a2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-29T19:26:26.231262Z",
     "iopub.status.busy": "2024-06-29T19:26:26.231060Z",
     "iopub.status.idle": "2024-06-29T19:26:26.233959Z",
     "shell.execute_reply": "2024-06-29T19:26:26.233570Z",
     "shell.execute_reply.started": "2024-06-29T19:26:26.231249Z"
    }
   },
   "source": [
    "The image can be dowloaded by using the left-hand file browser\n",
    "and navigating to the home directory, right-clicking on the file,\n",
    "and selecting \"Download\".\n",
    "\n",
    "If the left-hand file browser is not visible, in the main menu\n",
    "click \"View\" and then \"File Browser\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c89bd5f7-b9e1-48f7-b071-900a283d4fd2",
   "metadata": {},
   "source": [
    "#### 4.1.9. Display with afwDisplay\n",
    "\n",
    "If the image (with its header) has been saved locally, it\n",
    "can be read in as an `ExposureF` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb1307a4-dc04-4e3a-9b63-f88266adc93c",
   "metadata": {},
   "outputs": [],
   "source": [
    "calexp = ExposureF(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77836757-056d-4ba9-a0fd-a5c85c571937",
   "metadata": {},
   "source": [
    "The `ExposureF` format is the same format that images come in from the data butler.\n",
    "This format will work with the LSST Science Pipelines' `afwDisplay` package.\n",
    "\n",
    "Set `afwDisplay` to use `matplotlib`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9434c2e5-a530-4e89-afff-1b76e5e1988e",
   "metadata": {},
   "outputs": [],
   "source": [
    "afwDisplay.setDefaultBackend('matplotlib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb95172b-ba7e-4545-8a06-0e5e349c80d1",
   "metadata": {},
   "source": [
    "Display the image using `afwDisplay`, and center an orange circle on the target coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c02361f-a12c-47c8-a250-3a36b5d308e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "display = afwDisplay.Display(frame=fig)\n",
    "display.scale('linear', 'zscale')\n",
    "display.mtv(calexp.image)\n",
    "\n",
    "calexp_wcs = calexp.getWcs()\n",
    "targetPixels = calexp_wcs.skyToPixel(targetPoint)\n",
    "display.dot('o', targetPixels.getX(), targetPixels.getY(),\n",
    "            size=100, ctype='orange')\n",
    "del calexp_wcs, targetPixels\n",
    "\n",
    "plt.xlabel('x pixels')\n",
    "plt.ylabel('y pixels')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ff56d4b-0371-4074-9f21-26a6bf799479",
   "metadata": {},
   "source": [
    "Figure 3: The `calexp` displayed in grayscale, with different scaling compared to Figure 2. With `afwDisplay`, a color bar is shown by default, and the axes are in pixel coordinates. A large orange circle centered on the target coordinates appears as an arc in the lower-right corner, because the target coordinates are just off the edge of the `calexp`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6650597-d17a-4cba-80d3-9963ad633d42",
   "metadata": {},
   "source": [
    "Clean up before moving on to the next subsection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd3cc139-5d4f-4988-8b19-f16f1c806ecf",
   "metadata": {},
   "outputs": [],
   "source": [
    "del query, job, results\n",
    "del result, dl_results, image_url\n",
    "del filename, calexp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93eeffd2-d10f-4044-b525-b43d740f406c",
   "metadata": {},
   "source": [
    "### 4.2. deepCoadd\n",
    "\n",
    "Follow the same process as in Section 4.1, but instead of a\n",
    "`calexp` obtain a `deepCoadd` image.\n",
    "\n",
    "#### 4.2.1. Query, retrieve, and save\n",
    "\n",
    "Create the query for *i*-band `deepCoadd` images at 62, -37."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7479e4b-e93a-47cc-91fa-7863a2d99a88",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"SELECT * FROM ivoa.ObsCore \"\\\n",
    "        \"WHERE CONTAINS(POINT('ICRS', \" + \\\n",
    "        target_str_ra + \", \" + target_str_dec + \"), s_region) = 1 \"\\\n",
    "        \"AND lsst_band = 'i' \"\\\n",
    "        \"AND calib_level = 3 \"\\\n",
    "        \"AND dataproduct_subtype = 'lsst.deepCoadd_calexp'\"\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a6e8dfd-eb4d-448b-933e-12fa4fc6f535",
   "metadata": {},
   "source": [
    "Submit the job to the TAP service and fetch the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e702ec3-1a58-46a2-8e67-942d6dc1daaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "job = service.submit_job(query)\n",
    "job.run()\n",
    "job.wait(phases=['COMPLETED', 'ERROR'])\n",
    "print('Job phase is', job.phase)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11617ee6-59bf-4ac1-b840-8ef74829796e",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = job.fetch_result()\n",
    "print(len(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8dd8b5c-9697-4ca2-bd6e-40a592a2ebc8",
   "metadata": {},
   "source": [
    "Use the first image in the `results` table.\n",
    "Print the patch and tract."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70ab25f9-420c-4b84-aa5c-61ba43ee0889",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_0 = results[0]\n",
    "print(results_0['lsst_tract'])\n",
    "print(results_0['lsst_patch'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fffb59ab-209e-4b98-8c80-6dd109437d46",
   "metadata": {},
   "source": [
    "Save the image as a fits file in the home directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af9087cf-3a3f-4942-8a10-ce695f2c2030",
   "metadata": {},
   "outputs": [],
   "source": [
    "dl_results = DatalinkResults.from_result_url(results_0['access_url'],\n",
    "                                             session=auth_session)\n",
    "image_url = dl_results['access_url'][0]\n",
    "filename = os.path.join(os.getenv(\"HOME\"), 'dp02_02c_image_deepCoadd.fits')\n",
    "urlretrieve(image_url, filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7df4cf13-7368-43d7-9d7c-e89c4f5e0515",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-29T21:55:49.684402Z",
     "iopub.status.busy": "2024-06-29T21:55:49.684264Z",
     "iopub.status.idle": "2024-06-29T21:55:56.410772Z",
     "shell.execute_reply": "2024-06-29T21:55:56.409727Z",
     "shell.execute_reply.started": "2024-06-29T21:55:49.684391Z"
    }
   },
   "source": [
    "Load the `deepCoadd` image as an `ExposureF`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a13bb9a1-542d-4dad-a70e-6d1923d4879f",
   "metadata": {},
   "outputs": [],
   "source": [
    "deepCoadd = ExposureF(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "715082e0-9519-40bd-abad-d950f711e5e6",
   "metadata": {},
   "source": [
    "#### 4.2.2. Display with Firefly\n",
    "\n",
    "In Section 4.1, `matplotlib` was used to display the image with\n",
    "`imshow()` and via `afwDisplay`.\n",
    "\n",
    "As another example of image display, reset the backend of\n",
    "`afwDisplay` to be \"firefly\", which is the RSP's interactive\n",
    "image visualization tool\n",
    "\n",
    "Executing the following cell will open a new tab for the Firefly interface.\n",
    "\n",
    "**When Firefly opens:** click on its tab above, and drag it down towards the mid-right\n",
    "area of this interface for a side-by-side view."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1148a8ae-5e3b-42ac-a054-fb3584fa5e68",
   "metadata": {},
   "outputs": [],
   "source": [
    "afwDisplay.setDefaultBackend('firefly')\n",
    "afw_display = afwDisplay.Display(frame=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04b4b12d-2bb3-4c87-a5ba-6d5568ab72dc",
   "metadata": {},
   "source": [
    "Display the `deepCoadd` in Firefly, and set the mask plane to fully transparent\n",
    "so that only pixel data is visualized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8542dd2-44ed-4db3-98a0-791df09f8cf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "afw_display.mtv(deepCoadd)\n",
    "afw_display.setMaskTransparency(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10f5eec0-cdea-4bb5-a1a7-040c42ae252e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-29T21:56:14.505571Z",
     "iopub.status.busy": "2024-06-29T21:56:14.505117Z",
     "iopub.status.idle": "2024-06-29T21:56:14.507876Z",
     "shell.execute_reply": "2024-06-29T21:56:14.507438Z",
     "shell.execute_reply.started": "2024-06-29T21:56:14.505554Z"
    }
   },
   "source": [
    "Mark the query coordinates on the image as an orange circle.\n",
    "It will appear in the upper right corner of the image in Firefly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a4ee851-2ebd-4203-8e8f-4811c788a18c",
   "metadata": {},
   "outputs": [],
   "source": [
    "deepCoadd_wcs = deepCoadd.getWcs()\n",
    "targetPixel = deepCoadd_wcs.skyToPixel(targetPoint)\n",
    "afw_display.dot('o', targetPixel.getX(), targetPixel.getY(),\n",
    "                size=40, ctype='orange')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20664e1c-7916-4b74-baa3-c88aeb166bbe",
   "metadata": {},
   "source": [
    "To finish, clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7674e1e9-2f62-4edf-bf7f-3c801c53b5c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "del deepCoadd_wcs, targetPixel\n",
    "del query, job, results\n",
    "del results_0, dl_results, image_url\n",
    "del deepCoadd, filename"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d31e7a96-57a6-439d-81e4-f010c1825709",
   "metadata": {},
   "source": [
    "## 5. Exercises for the learner\n",
    "\n",
    "(1) Follow the process in Section 4.1 to retrieve and visualize a difference image.\n",
    "\n",
    "(2) Use the `ccdvisitid` to obtain all detected sources in a `calexp`. Use the TAP service to retrieve them and plot them onto the calexp. (Alternatively, use the tract and patch to obtain all detect objects in a `deepCoadd`.)\n",
    "\n",
    "Hint for (2):\n",
    "```\n",
    "SELECT ccdVisitId,coord_dec,coord_ra\n",
    "FROM dp02_dc2_catalogs.Source\n",
    "WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec),CIRCLE('ICRS', 62, -37, 0.25))=1\n",
    "AND (ccdVisitId = 214437078)```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "908f0e90-a1ff-43bb-8e86-89534dd5d119",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}