FeatureFinderNotebook_v1.2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# _Herschel_ SPIRE Spectral Feature Finder\n",
    "\n",
    "## A notebook with simple examples on using the Spectral Feature Finder products\n",
    "\n",
    "**Authors:** Jeremy Scott (1), Ivan Valtchanov (2), Christopher Benson (1) and Locke Spencer (1).\n",
    "\n",
    "(1) University of Lethbridge, Dr. Locke Spencer's Lab, E736, Alberta, Canada </br>\n",
    "(2) Herschel Science Centre, European Space Astronomy Centre, European Space Agency\n",
    "\n",
    "**Version:** 1.0\n",
    "\n",
    "**Date:** 02 Apr 2019"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "\n",
    "The Herschel/SPIRE Spectral Feature Finder (FF) is an automated tool which detects significant spectral features within SPIRE Fourier Transform Spectrometer (FTS) spectral data. The FF is designed to facilitate data-mining and preliminary analysis of SPIRE spectral data and is not a replacement for detailed scientific analysis. A full introduction to the FF routine and products is presented at http://archives.esac.esa.int/hsa/legacy/HPDP/SPIRE/SPIRE-S/spectral_feature_catalogue/SpectralFeatureFinder_releaseNote.html The FF products can be accessed from https://www.cosmos.esa.int/web/herschel/spire-spectral-feature-catalogue. In this notebook we provide examples of how to search FF products for spectral lines or sources on interest, and include sample code for visually displaying the results and estimating integrated line flux. \n",
    "\n",
    "The full FF products can be downloaded from the link above, however, in this notebook we will only download select products when necessary. Products will be downloaded from the `archive` defined below and will be placed in a chosen directory which we will assigned to the variable `dataDir`. The `archive` has a directory structure identical to the FF products you can download from the link above.\n",
    "\n",
    "To begin, a few Python modules will need to be imported. The standard Anaconda distribution is recommended as it includes most of these packages by default. However, Anaconda does not ship with the spectral-cube or lmfit packages and likely need to be installed by the user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requirements\n",
    "\n",
    "This notebook was developed and needs python 3.7 and the necessary packages are listed in [environment.yml](environment.yml) file. You can use the file with conda:\n",
    "\n",
    "```\n",
    "conda env create -f environment.yml\n",
    "```\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import io\n",
    "import requests\n",
    "import gzip\n",
    "import csv\n",
    "import tarfile\n",
    "import urllib3\n",
    "http = urllib3.PoolManager()\n",
    "\n",
    "import numpy as np\n",
    "from lmfit import Model\n",
    "\n",
    "from astropy import units as u\n",
    "from astropy.wcs import WCS\n",
    "from astropy.io import fits\n",
    "from astropy.coordinates import SkyCoord\n",
    "from astropy.table import Table\n",
    "\n",
    "from spectral_cube import SpectralCube\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the URL for the FTS Spectral Feature Catalogue\n",
    "archive = 'http://archives.esac.esa.int/hsa/legacy/HPDP/SPIRE/SPIRE-S/spectral_feature_catalogue'\n",
    "\n",
    "# the work folder where the data will be downloaded. It will be used as a simple cache folder too.\n",
    "dataDir = os.path.join(os.path.expanduser('~'),'tmp/HerschelData')\n",
    "if (not os.path.isdir(dataDir)):\n",
    "    print (f'{dataDir} does not exist!')\n",
    "    print ('Please set dataDir to point to an already existing folder')\n",
    "else:\n",
    "    print (f\"INFO: all temprary files and downloads will be stored in {dataDir}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching the Catalogue\n",
    "\n",
    "Searching the features catalogue for features meeting a specific search criteria is done most efficiently with the the SPIRE Automated Feature Extraction Catalogue (SAFECAT). The SAFECAT is a fits table which contains a row entry for each feature detected by the FF. Below we demonstrate how to retrieve and search the SAFECAT for detected features within a user-defined frequency range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SAFECAT = Table.read(f'{archive}/SpireAutomatedFeatureExtractionCATalogue/SAFECAT_v2_restFrame.fits.gz')\n",
    "SAFECAT.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# frequency range\n",
    "freqRange = (806.0, 809.0)  #GHz\n",
    "restFrame = True\n",
    "\n",
    "column = 'frequency'\n",
    "if restFrame:\n",
    "    column = 'restFrameFreq'\n",
    "\n",
    "ind = np.where((SAFECAT[column] <= max(freqRange)) & (SAFECAT[column] >= min(freqRange)))[0]\n",
    "out = SAFECAT[ind]\n",
    "out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `out` is a table extracted from the full SAFECAT with all entries that match the frequency search range. Queries using other search criteria can be accomplished using a similar format. Below we demonstrate queries based on radial velocity and observation identification number (obsid)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# velociy range\n",
    "velRange = (1000.0, 1020.0) #km/s\n",
    "column = 'velocity'\n",
    "\n",
    "ind = np.where((SAFECAT[column] <= max(velRange)) & (SAFECAT[column] >= min(velRange)))[0]\n",
    "print(f\"Found {len(ind)} features within a velocity range of {velRange} km/s\")\n",
    "#out = SAFECAT[ind]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# obsid\n",
    "obsid = 1342243604\n",
    "column = 'obsid'\n",
    "\n",
    "ind = np.where(SAFECAT[column] == obsid)[0]\n",
    "print(f\"Found {len(ind)} features with OBSID {obsid}\")\n",
    "#out = SAFECAT[ind]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a final example, we demonstrate how to query results based on sky coordinates. Herschel uses the equatorial coordinate system with right ascension (RA) and declination (DEC) recorded in units of degrees. Converting from HH:MM:SS to degrees can be accomplished easily using the `SkyCoord` method in the astropy module (see documentation for proper implementation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# coordinates\n",
    "ra_in = '21 7 1.7' # HH:MM:SS\n",
    "dec_in = '42 14 11' # DD:MM:SS\n",
    "\n",
    "# conversion\n",
    "c = SkyCoord(ra_in, dec_in, frame='icrs', unit=(u.hourangle, u.deg))\n",
    "\n",
    "# search radius\n",
    "radius = 0.5 * u.arcmin #arcmin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use astropy.coordinates methods for catalogue matching and separarion\n",
    "catalog = SkyCoord(SAFECAT['ra'],SAFECAT['dec'],frame='icrs',unit=(u.deg,u.deg))\n",
    "d2d = c.separation(catalog)\n",
    "in_radius = d2d <= radius\n",
    "ind = np.where(in_radius)[0]\n",
    "\n",
    "print (f\"Found {len(ind)} features within a radius of {radius} about {c}\")\n",
    "out = SAFECAT[ind]\n",
    "obsids = np.unique(out['obsid'])\n",
    "print(f\"Number of unique obsids: {len(obsids)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visual Inspection\n",
    "The FF products include postcards for quick inspection of the FF results. These postcards can be opened using Python as demonstrated below. To do this efficiently, one must know the obsid, calibration scheme, and observation mode for the source of interest. This information is encoded in the SAFECAT, and by extension the `out` sub-table using the column headings/keys \"obsid\", \"calibration\", and \"sampling\", respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "obsid = 1342191234\n",
    "column = 'obsid'\n",
    "\n",
    "ind = np.where(SAFECAT[column] == obsid)[0]\n",
    "out = SAFECAT[ind]\n",
    "\n",
    "cal = out['calibration'][0] # options: 'extended', 'pointSource'\n",
    "obs_mod = out['sampling'][0] # options: 'full', 'sparse'\n",
    "hpdp = int(out['hpdp'][0])\n",
    "bgs = int(out['bgs'][0])\n",
    "specType = ''\n",
    "if hpdp:\n",
    "    specType = 'HPDP'\n",
    "elif bgs:\n",
    "    specType = 'BGS'\n",
    "else:\n",
    "    specType = 'SPG'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getPaths(obsid, cal, obs_mod):\n",
    "    \"\"\"\n",
    "    utility def to return the URLs for the postcard, the features catalogue and the continuum file\n",
    "    \"\"\"\n",
    "    pntDir = f'{archive}/HRpointProducts' # directory for point source calibrated sparse observations\n",
    "    extDir = f'{archive}/HRextProducts' # directory for extended source calibrated sparse observations\n",
    "    mapDir = f'{archive}/HRmapping' # directory for extended source calibrated full sampling observations (mapping)\n",
    "    if ('full' in obs_mod or 'intermediate' in obs_mod) and ('extended' in cal):\n",
    "        pFile = f'{mapDir}/postcards/{obsid}_postcard_comb_2x3.png'\n",
    "        obsFile = f'{mapDir}/featureCatalogues/{obsid}_featuresFound.fits'\n",
    "        contFile = f'{mapDir}/continuumParameters/{obsid}_fittedContinuumParameters.fits'\n",
    "    elif ('sparse' in obs_mod) and (cal == 'pointSource'):\n",
    "        # PNG postcards are in a hidden folder\n",
    "        pFile = f'{archive}/.images/postcards_png/{obsid}_postcard_pnt.png'\n",
    "        obsFile = f'{pntDir}/featureCatalogues/{obsid}_featuresFound_pnt.fits'\n",
    "        contFile = f'{pntDir}/continuumParameters/{obsid}_fittedContinuumParameters_pnt.fits'\n",
    "    elif ('sparse' in obs_mod) and ('extended' in cal):\n",
    "        pFile = f'{archive}/.images/postcards_png/{obsid}_postcard_ext.png'\n",
    "        obsFile = f'{extDir}/featureCatalogues/{obsid}_featuresFound_ext.fits'\n",
    "        contFile = f'{extDir}/continuumParameters/{obsid}_fittedContinuumParameters_ext.fits'\n",
    "    else:\n",
    "        print('Invalid inputs')\n",
    "        pFile = None\n",
    "        obsFile = None\n",
    "        contFile = None\n",
    "        \n",
    "    # now check if those files are available\n",
    "    r = http.request('HEAD',pFile)\n",
    "    if (r.status != 200):\n",
    "        print (\"Warning, postcard file {pFile} not found\")\n",
    "        pFile = None\n",
    "    r = http.request('HEAD',obsFile)\n",
    "    if (r.status != 200):\n",
    "        print (\"Warning, featureFinder table {obsFile} not found\")\n",
    "        obsFile = None\n",
    "    r = http.request('HEAD',contFile)\n",
    "    if (r.status != 200):\n",
    "        print (\"Warning, continuum file {contFile} not found\")\n",
    "        contFile = None\n",
    "    #\n",
    "    return pFile, obsFile, contFile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pURL, obsURL, contURL = getPaths(obsid, cal, obs_mod)\n",
    "print(f\"Postcard location: {pURL}\")\n",
    "img = plt.imread(pURL)\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.imshow(img)\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `obsURL` and `contURL` variables point to the files containing the features catalogue and fitted continuum parameters for the specified observation. These will be used later.\n",
    "\n",
    "It is also possible to download SPIRE spectral data directly from the Herschel Science Archive (HSA) using Python. These data can then be plotted along with features detected by the FF meeting a search criteria. Manipulating SPIRE spectral cubes is accomplished using the \"spectral-cube\" package of astropy, which may need to be installed by the user. SPIRE spectral data can be retrieved using the function `getSpireFtsData` defined below. \n",
    "\n",
    "Note that the `getSpireFtsData` function returns both \"naive\" and \"convolved\" hyper-spectral cubes, and has options for both \"apodized\" and \"unapodized\" spectra. FF products, however, are only available for unapodized spectra. Additionally, for sparse observations, the `getSpireFtsData` function is configured to return spectra from only the central \"SLWC3\" and \"SSWD4\" detectors. For sparse observations, FF products are only available for unapodized spectra from the central detectors. \n",
    "\n",
    "It should also be noted that many sparse observations processed by the FF use Highly Processed Data Products (HPDPs; http://archives.esac.esa.int/hsa/legacy/HPDP/SPIRE/SPIRE-S/cal_targets/) or Background Subtracted data (BGS; http://archives.esac.esa.int/hsa/legacy/HPDP/SPIRE/SPIRE-S/BKGS/). These spectra are stored in a different location that the Standard Product Generation (SPG) spectra. To retrieve to spectrum processed by the FF, the appropriate data type (`specType`) must be specified in the `getSpireFtsData` function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getSpireFtsData(obsid, cal, obs_mod, specType, apod=False, cache=True, tmpDir=\"\"):\n",
    "    \"\"\"\n",
    "    Using the HTTP access to HAIO, retrieve the level-2 products in a tar file\n",
    "    and extract only the requested fits files\n",
    "    \n",
    "    cubeType can be 'convol' or 'naive'\n",
    "    apod controls whether to extract the apodized version (if True)\n",
    "    \n",
    "    simple caching is impemented, checking of a tar file with the same name (OBSID based) already exists\n",
    "    at the default folder.\n",
    "    \"\"\"\n",
    "    \n",
    "    spg_url = \"http://archives.esac.esa.int/hsa/aio/jsp/\"\n",
    "    hpdp_url = \"http://archives.esac.esa.int/hsa/legacy/HPDP/SPIRE/SPIRE-S/cal_targets/SPIREspec_calibrators/\"\n",
    "    bgs_url = \"http://archives.esac.esa.int/hsa/legacy/HPDP/SPIRE/SPIRE-S/BKGS/spectra/\"\n",
    "    \n",
    "    if obs_mod != 'full':\n",
    "        detectors = ['SLWC3', 'SSWD4']\n",
    "        specDets = {det:[] for det in detectors}\n",
    "        headers = {det:[] for det in detectors}\n",
    "    if specType == 'SPG':\n",
    "        vers = '_spg_'\n",
    "        if (apod): \n",
    "            vers = '_spgApod_'\n",
    "        tarFile = tmpDir + \"/%i_FTS_level2.tar\"%obsid\n",
    "        if (os.path.isfile(tarFile) and cache):\n",
    "            print (\"Found an already existing tar file for OBSID %i. Will Use it\"%obsid)\n",
    "        else:\n",
    "            haioRequest = spg_url + \"product.jsp?PROTOCOL=HTTP&OBSERVATION_ID=%i&PRODUCT_LEVEL=Level2\"%obsid\n",
    "            print (\"Downloading level-2 data from the Herschel Science Archive. May take a while... be patient\")\n",
    "            r = requests.get(haioRequest)\n",
    "            with open(tarFile, \"wb\") as tmp:\n",
    "                tmp.write(r.content)\n",
    "        # now read the downloaded tar file.\n",
    "        with tarfile.open(tarFile,'r') as tar:\n",
    "            if obs_mod == 'full': # mapping observations \n",
    "                cube = {}\n",
    "                headers = {}\n",
    "                for member in tar.getmembers():\n",
    "                    if (('_20ssc_' in member.name) and (vers in member.name)):\n",
    "                        ctype = 'naive'\n",
    "                        if ('convol' in member.name):\n",
    "                            ctype = 'convol'\n",
    "                        f=tar.extractfile(member)\n",
    "                        xx = fits.open(gzip.open(io.BytesIO(f.read())))\n",
    "                        detx = xx[0].header['DETECTOR']\n",
    "                        res = xx[0].header['PROC_RES']\n",
    "                        # comined key for the output dictionary\n",
    "                        xkey = \"%s_%s_%s\"%(detx,res,ctype)\n",
    "                        cubeData = xx['image'].data\n",
    "                        header = xx['image'].header\n",
    "                        header[\"CTYPE3\"] = \"FREQ\"\n",
    "                        cube[xkey]= SpectralCube(data=cubeData, wcs=WCS(header))\n",
    "                        headers[xkey] = xx[0].header\n",
    "                return cube, headers, header\n",
    "            elif obs_mod == 'sparse': # sparse observations\n",
    "                calDict = {'pointSource':'point', 'extended':'ext'} \n",
    "                for member in tar.getmembers():\n",
    "                    if (('_20s' in member.name) and (vers in member.name) and (calDict[cal] in member.name)):\n",
    "                        f=tar.extractfile(member)\n",
    "                        xx = fits.open(gzip.open(io.BytesIO(f.read())))\n",
    "    else:\n",
    "        if specType == 'HPDP':\n",
    "            r = requests.get(hpdp_url)\n",
    "            url = hpdp_url\n",
    "        elif specType == 'BGS':\n",
    "            r = requests.get(bgs_url)\n",
    "            url = bgs_url\n",
    "        else:\n",
    "            print(\"Invalid specType\")\n",
    "            return False, False\n",
    "        i_start = int(r.text.find(str(obsid)))\n",
    "        if i_start == -1:\n",
    "            print(\"No %s for observation %i\"%(specType,obsid))\n",
    "            return False, False\n",
    "        i_end = i_start + int(r.text[i_start:].find('gz')) + 2\n",
    "        gzFile = tmpDir + \"/\" + r.text[i_start:i_end]\n",
    "        if (os.path.isfile(gzFile) and cache):\n",
    "            print (\"Found an already existing gz file for OBSID %i. Will Use it\"%obsid)\n",
    "        else:\n",
    "            haioRequest = url + r.text[i_start:i_end]\n",
    "            print (\"Downloading %s data from the Archive. May take a while... be patient\"%specType)\n",
    "            r = requests.get(haioRequest)\n",
    "            with open(gzFile, \"wb\") as tmp:\n",
    "                tmp.write(r.content)\n",
    "        xx = fits.open(gzip.open(gzFile))\n",
    "    header0 = xx[0].header\n",
    "    for det in detectors:\n",
    "        specDets[det] = xx[det].data\n",
    "        headers[det] = xx[det].header\n",
    "    return specDets, header0, headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sparse FF Producs and SPIRE Spectral Datasets\n",
    "\n",
    "Currently our code is pointed at the observation 1342191234 which is a point source calibrated sparse observation. To retrieve the SPIRE level-2 spectral data for this observation, we call the `getSpireFtsData` function as demonstrated below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "specData, header0, headers = getSpireFtsData(obsid, cal, obs_mod, specType, apod=False, cache=True, tmpDir=dataDir)\n",
    "\n",
    "print(f\"\\nData keys: {specData.keys()}\")\n",
    "print(f\"Header keys: {headers.keys()}\")\n",
    "print(f\"Target: {header0['OBJECT']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen, the \"specData\" and \"headers\" dictionaries contain entries for both central detectors. We will focus primarily on the frequency and flux axes of the spectral data. For more details about the data structure of SPIRE observations, the reader is referred to [the SPIRE data reduction guide](http://herschel.esac.esa.int/hcss-doc-15.0/load/spire_drg/html/spire_drg.html).\n",
    "\n",
    "With the spectral data in hand, we can now generate a plot of the spectral data and include the fitted continuum and additional lines to indicate spectral features detected by the FF. In principle, we could search the SAFECAT for the desired features as demonstrated above. This time, however, we will extract the desired lines from the features catalogue specific to the 1342191234 observation. The location of the observation specific features catalogue and its fitted continuum parameters are currently stored in the `obsFile` and `contFile` variables, respectively, defined above. When plotting lines to indicated the position of desired features, these lines will be proportional to the associated SNRs of the features, much like the lines in the FF postcards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# retrieve observation specific catalogue and continuum, directly from the URL\n",
    "data = Table.read(obsURL)\n",
    "data.info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cont = Table.read(contURL)\n",
    "cont.info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unpack data for convenience\n",
    "features = data['frequency']\n",
    "snr = data['SNR']\n",
    "detector = data['detector']\n",
    "velocity = data.meta['RV']\n",
    "velocity_error = data.meta['RV_ERR']\n",
    "\n",
    "# select lines of interest\n",
    "freqRange = (900.0, 1500.0)\n",
    "ind = np.where((features >= min(freqRange)) & (features <= max(freqRange)))[0]\n",
    "print (f\"Found {len(ind)} features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate fitted continuum\n",
    "x_slw, y_slw = specData['SLWC3']['wave'], specData['SLWC3']['flux']\n",
    "x_ssw, y_ssw = specData['SSWD4']['wave'], specData['SSWD4']['flux']\n",
    "cont_slw = cont['p0'][0] + cont['p1'][0]*x_slw + cont['p2'][0]*x_slw**2 + cont['p3'][0]*x_slw**3\n",
    "cont_ssw = cont['p0'][1] + cont['p1'][1]*x_ssw + cont['p2'][1]*x_ssw**2 + cont['p3'][1]*x_ssw**3\n",
    "flux_range = (max([max(y_slw),max(y_ssw)]) - min([min(y_slw),min(y_ssw)]))\n",
    "offset = flux_range*0.05\n",
    "scale = flux_range/max(snr)/2.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plotting\n",
    "fig = plt.figure(0, figsize=(10,6))\n",
    "plt.clf()\n",
    "plt.title(f\"SPIRE Observation {obsid}\\nRV = {velocity:.2f} $\\\\pm$ {velocity_error:.2f} [km/s]\", fontsize=14)\n",
    "plt.plot(x_slw, y_slw, 'r', label='SLWC3')\n",
    "plt.plot(x_ssw, y_ssw, 'b', label='SSWD4')\n",
    "plt.plot(x_slw, cont_slw, 'g', linewidth=2.0, label='continuum')\n",
    "plt.plot(x_ssw, cont_ssw, 'g', linewidth=2.0)\n",
    "for i in ind:\n",
    "    freq = features[i]\n",
    "    sn = snr[i]*scale\n",
    "    det = detector[i]\n",
    "    base = specData[det]['flux'][np.where(min(abs(specData[det]['wave'] - freq)) == \n",
    "                                          abs(specData[det]['wave'] - freq))[0]]\n",
    "    plt.plot([freq, freq], [base+offset, base+offset+sn], 'k')\n",
    "limits = plt.ylim()\n",
    "plt.plot([min(freqRange), min(freqRange)],[min(limits), max(limits)], '--k')\n",
    "plt.plot([max(freqRange), max(freqRange)],[min(limits), max(limits)], '--k')\n",
    "plt.ylim(limits)\n",
    "plt.xlabel(f\"{headers['SLWC3'].comments['TTYPE1']} [{headers['SLWC3']['TUNIT1']}]\", fontsize=14)\n",
    "plt.ylabel(f\"{headers['SLWC3'].comments['TTYPE2']} [{headers['SLWC3']['TUNIT2']}]\", fontsize=14)\n",
    "plt.legend(loc=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only features at frequencies in (900,1500) are shown as this was the selection criteria earlier on.\n",
    "\n",
    "### Non-standard observations\n",
    "\n",
    "Note that observations 1342227785 and 1342227778 are treated somewhat differently than other sparse observations in the FF products due to their irregular spatial sampling (see release notes). As such, users must be careful when accessing these data from the HSA, and the correct prefix `100` or `200` for beam steering mirror position 1 and 2, respectively, must be added to the obsid when searching for the corresponding postcards."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mapping FF Producs and SPIRE Spectral Datasets\n",
    "\n",
    "We now turn to mapping observations. SPIRE data for mapping observations is packaged into \"spectral cubes\" (or hyper-spectral cubes) which are 3-dimensional data structures. Two axes correspond to pixel coordinates which represent different locations on-sky. The third axis corresponds to the spectral data. A pixel with spectral dimension is sometimes called _spaxel_.\n",
    "\n",
    "Various cubes for the same mapping observation can be accessed from the HSA and retrieved using the `getSpireFtsData` function. FF products, however, are only available for High Resolution (HR) unapodized SLW and SSW convolved cubes. Details about the various cubes can be found in [the SPIRE data reduction guide](http://herschel.esac.esa.int/hcss-doc-15.0/load/spire_drg/html/spire_drg.html).\n",
    "\n",
    "We begin by reassigning some variables, and retrieving the SPIRE data for a mapping observation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "obsid = 1342192174\n",
    "cal = 'extended'\n",
    "obs_mod = 'full'\n",
    "specType = 'SPG'\n",
    "\n",
    "pURL, obsURL, contURL = getPaths(obsid, cal, obs_mod)\n",
    "print(f\"Postcard location: {pURL}\")\n",
    "img = plt.imread(pURL)\n",
    "plt.figure(figsize=(10,6))\n",
    "plt.imshow(img)\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "specData, header0, headers = getSpireFtsData(obsid, cal, obs_mod, specType, apod=False, cache=True, tmpDir=dataDir)\n",
    "cube_slw, cube_ssw = specData['SLW_HR_convol'], specData['SSW_HR_convol']\n",
    "header_slw, header_ssw = cube_slw.header, cube_ssw.header\n",
    "d_slw, d_ssw = header_slw['CDELT2'], header_ssw['CDELT2']  # pixel widths of SLW and SSW maps, respectively\n",
    "for kk in header0.keys(): target = header0[kk][\"OBJECT\"] ; break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "slw_shape = cube_slw.shape\n",
    "ssw_shape = cube_ssw.shape\n",
    "print (f'Target: {target}')\n",
    "print(f\"SLW Cube \\n Shape: {slw_shape}; {slw_shape[0]} Frequency points, {slw_shape[1]} Rows, {slw_shape[2]} Columns\", \\\n",
    "       f\"\\n Pixel width: {d_slw*3600:.1f} [arcsec]\")\n",
    "print(f\"SSW Cube \\n Shape: {ssw_shape}; {ssw_shape[0]} Frequency points, {ssw_shape[1]} Rows, {ssw_shape[2]} Columns\", \\\n",
    "       f\"\\n Pixel width: {d_ssw*3600:.1f} [arcsec]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be inferred from the dimensions of the SLW and SSW maps, in addition to their respective pixel widths, the SLW and SSW maps do not occupy the same footprint on-sky. This disparity can be easily seen in the mapping postcard. As such, one needs to take care when selecting SLW and SSW spaxels if one wishes to inspect the spectra form the same location on-sky with both maps. \n",
    "\n",
    "Below we demonstrate how to find the nearest SLW spaxel given an initial SSW spaxel using the FF products. We perform this operation with the observation specific features catalogue but the process in nearly identical when performed on the SAFECAT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# extract data from observation specific features catalogue\n",
    "# cannot use Table for multi-extension tables\n",
    "hdulist = fits.open(obsURL)\n",
    "hdulist.info()\n",
    "FF_data = hdulist[1].data\n",
    "velocity_map = hdulist[2].data\n",
    "header = hdulist[1].header\n",
    "hdulist.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reference SSW pixel (SPIRE data)\n",
    "c_col = int(header_ssw['CRPIX1'])-1\n",
    "c_row = int(header_ssw['CRPIX2'])-1\n",
    "c_ra, c_dec = header_ssw['CRVAL1'], header_ssw['CRVAL2']\n",
    "pix_coord = SkyCoord(c_ra, c_dec, frame='icrs', unit=(u.deg, u.deg))\n",
    "print(f\"SSW pixel {c_row, c_col} has coordinates {pix_coord}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get indicies for all entries from reference SSW pixel (FF products)\n",
    "ssw_ind = np.where(FF_data['array'] == 'SSW')[0]\n",
    "ssw_coord = SkyCoord(FF_data['ra'][ssw_ind], FF_data['dec'][ssw_ind], frame='icrs', unit=(u.deg, u.deg))\n",
    "d2d = pix_coord.separation(ssw_coord)\n",
    "ssw_match = np.where(abs(d2d.deg) <= d_ssw/10.0) # not using '==' because of potential rounding \n",
    "\n",
    "ssw_ra, ssw_dec = np.unique(FF_data['ra'][ssw_ind][ssw_match]), np.unique(FF_data['dec'][ssw_ind][ssw_match])\n",
    "ssw_row, ssw_col = np.unique(FF_data['row'][ssw_ind][ssw_match]), np.unique(FF_data['column'][ssw_ind][ssw_match])\n",
    "\n",
    "# check that we are looking at the same on-sky location\n",
    "if (min(abs(ssw_ra - c_ra)) > d_ssw/2.0) or (min(abs(ssw_dec - c_dec)) > d_ssw/2.0):\n",
    "    print('SPIRE data and FF product coordinates do NOT match within half resolution of SSW mapping pixel.')\n",
    "    print(f\"FF Data: RA = {ssw_ra} Dec = {ssw_dec}\")\n",
    "    print(f\"SPIRE: RA = {c_ra} Dec = {c_dec}\")\n",
    "else:\n",
    "    print('SPIRE data and FF product coordinates DO match within half resolution of SSW mapping pixel.')\n",
    "    print(f\"FF SSW pixel {*ssw_row, *ssw_col} has coordinates {*ssw_ra, *ssw_dec} [deg]\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get indicies for all entries from SLW pixel nearest central SSW pixel (FF products)\n",
    "slw_ind = np.where(FF_data['array'] == 'SLW')[0]\n",
    "slw_coord = SkyCoord(FF_data['ra'][slw_ind], FF_data['dec'][slw_ind], frame='icrs', unit=(u.deg, u.deg))\n",
    "d2d = pix_coord.separation(slw_coord)\n",
    "slw_match = np.where((min(abs(d2d.deg)) == abs(d2d.deg)) & (abs(d2d.deg) <= np.sqrt(2)*d_slw))\n",
    "\n",
    "slw_ra, slw_dec = np.unique(FF_data['ra'][slw_ind][slw_match]), np.unique(FF_data['dec'][slw_ind][slw_match])\n",
    "slw_row, slw_col = np.unique(FF_data['row'][slw_ind][slw_match]), np.unique(FF_data['column'][slw_ind][slw_match])\n",
    "\n",
    "print(f\"\\n     row  column\\nSLW {slw_row[0]:4} {slw_col[0]:4}\\nSSW {ssw_row[0]:4} {ssw_col[0]:4}\")\n",
    "print(f\"\\nFF coordinates of central SSW pixel = ({ssw_ra[0]:.4f}, {ssw_dec[0]:.4f}) [deg]\")\n",
    "print(f\"FF coordinates of nearest SLW pixel = ({slw_ra[0]:.4f}, {slw_dec[0]:.4f}) [deg]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen from the above printout, we have successfully located the SLW pixel nearest the central SSW pixel.\n",
    "\n",
    "Having determined which SLW and SSW pixels we wish to inspect, we can now generate a plot similar to the one generated for our example sparse observation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# organize FF results \n",
    "features, snr, array = [], [], []\n",
    "#\n",
    "features.extend(FF_data['frequency'][slw_ind][slw_match])\n",
    "snr.extend(FF_data['snr'][slw_ind][slw_match])\n",
    "array.extend(FF_data['array'][slw_ind][slw_match])\n",
    "#\n",
    "features.extend(FF_data['frequency'][ssw_ind][ssw_match])\n",
    "snr.extend(FF_data['snr'][ssw_ind][ssw_match])\n",
    "array.extend(FF_data['array'][ssw_ind][ssw_match])\n",
    "#\n",
    "features = np.array(features)\n",
    "slw_velocity, ssw_velocity = np.mean(FF_data['velocity'][slw_ind][slw_match]), np.mean(FF_data['velocity'][ssw_ind][ssw_match])\n",
    "slw_vel_error, ssw_vel_error = np.mean(FF_data['velocityError'][slw_ind][slw_match]), np.mean(FF_data['velocityError'][ssw_ind][ssw_match])\n",
    "# organize SPIRE data\n",
    "# this two lines work for spectral_cube version 0.4.3 or above\n",
    "x_slw, y_slw = cube_slw.spectral_axis.value/1.0e9, cube_slw.unmasked_data[:,slw_row[0],slw_col[0]].value\n",
    "x_ssw, y_ssw = cube_ssw.spectral_axis.value/1.0e9, cube_ssw.unmasked_data[:,ssw_row[0],ssw_col[0]].value\n",
    "#\n",
    "# for spectral_cube versions earlier than 0.4.3\n",
    "#x_slw, y_slw = cube_slw.spectral_axis.value/1.0e9, cube_slw[:,slw_row[0],slw_col[0]].value\n",
    "#x_ssw, y_ssw = cube_ssw.spectral_axis.value/1.0e9, cube_ssw[:,ssw_row[0],ssw_col[0]].value\n",
    "flux = {'SLW':y_slw, 'SSW':y_ssw}\n",
    "wave = {'SLW':x_slw, 'SSW':x_ssw}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# retrieve and generate the continua\n",
    "hdulist = fits.open(contURL)\n",
    "cont_data = hdulist[1].data\n",
    "hdulist.close()\n",
    "\n",
    "cont_slw_ind = np.where((cont_data['row'] == slw_row[0]) & (cont_data['column'] == slw_col[0]) & (cont_data['array'] == 'SLW'))[0]\n",
    "cont_ssw_ind = np.where((cont_data['row'] == ssw_row[0]) & (cont_data['column'] == ssw_col[0]) & (cont_data['array'] == 'SSW'))[0]\n",
    "\n",
    "cont_data_slw = {key:vals for (key, vals) in zip(cont_data.names, cont_data[cont_slw_ind[0]])}\n",
    "cont_data_ssw = {key:vals for (key, vals) in zip(cont_data.names, cont_data[cont_ssw_ind[0]])}\n",
    "\n",
    "cont_slw = cont_data_slw['p0'] + cont_data_slw['p1']*x_slw + cont_data_slw['p2']*x_slw**2 + cont_data_slw['p3']*x_slw**3 \n",
    "cont_ssw = cont_data_ssw['p0'] + cont_data_ssw['p1']*x_ssw + cont_data_ssw['p2']*x_ssw**2 + cont_data_ssw['p3']*x_ssw**3\n",
    "#\n",
    "flux_range = (max([max(y_slw),max(y_ssw)]) - min([min(y_slw),min(y_ssw)]))\n",
    "offset = flux_range*0.05\n",
    "scale = flux_range/max(snr)/2.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select lines of interest\n",
    "freqRange = (600.0, 1500.0)\n",
    "ind = np.where((features >= min(freqRange)) & (features <= max(freqRange)))[0]\n",
    "\n",
    "# plotting\n",
    "fig = plt.figure(0, figsize=(10,6))\n",
    "plt.clf()\n",
    "plt.title(f\"SPIRE Observation {obsid}\\nRV = {np.mean([slw_velocity, ssw_velocity]):.2f}\" \\\n",
    "          f\"$\\\\pm$ {np.mean([slw_vel_error, ssw_vel_error]):.2f} [km/s]\", fontsize=14)\n",
    "plt.plot(x_slw, y_slw, 'r', label=f'SLW spaxel ({slw_row[0]},{slw_col[0]})')\n",
    "plt.plot(x_ssw, y_ssw, 'b', label=f'SSW spaxel ({ssw_row[0]},{ssw_col[0]})')\n",
    "plt.plot(x_slw, cont_slw, 'g', linewidth=2.0, label='continuum')\n",
    "plt.plot(x_ssw, cont_ssw, 'g', linewidth=2.0)\n",
    "for i in ind:\n",
    "    freq = features[i]\n",
    "    sn = snr[i]*scale\n",
    "    ary = array[i]\n",
    "    base = flux[ary][np.where(min(abs(wave[ary] - freq)) == abs(wave[ary] - freq))[0]]\n",
    "    plt.plot([freq, freq], [base+offset, base+offset+sn], 'k')\n",
    "limits = plt.ylim()\n",
    "plt.plot([min(freqRange), min(freqRange)],[min(limits), max(limits)], '--k')\n",
    "plt.plot([max(freqRange), max(freqRange)],[min(limits), max(limits)], '--k')\n",
    "plt.ylim(limits)\n",
    "plt.xlabel('Frequency [GHz]', fontsize=14)\n",
    "plt.ylabel('Brightness [$\\\\mathrm{W \\; m^{-2} \\; Hz^{-1} \\; sr^{-1}}$]', fontsize=14)\n",
    "plt.legend(loc=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Velocity Maps\n",
    "\n",
    "For each spectrum processed by the FF, an attempt was made to estimate source radial velocity. These estimates were generated using a variety of methods including: pattern searching for $^{12}$CO rotational emission features, cross-correlation methods where detected features are compared against a template of lines commonly found within the FIR band, and using external references. For more details, please see the release notes linked in the introduction of this document. \n",
    "\n",
    "For point-source observations, velocity estimates are stored in the meta data of the per-obsid FF products. For mapping observations, velocity estimates are stored in a velocity map table associated with the per-obsid mapping observation products. Additionally, the main table in each per-obsid FF mapping product contains a column storing radial velocity estimates on a per-feature basis. The SAFECAT also contains a column presenting a radial velocity estimate for each detected feature for the entire FF dataset. In the sample code below, we demonstrate how to generate a velocity map for the already loaded mapping observation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# retrieve World Coordinate System (WCS) object from SPIRE HSA observation\n",
    "wcs_hsa = WCS(header_ssw)\n",
    "wcs_new = WCS(naxis=2)\n",
    "\n",
    "wcs_new.wcs.crpix = wcs_hsa.wcs.crpix[:2]\n",
    "wcs_new.wcs.cdelt = wcs_hsa.wcs.cdelt[:2]\n",
    "wcs_new.wcs.crval = wcs_hsa.wcs.crval[:2]\n",
    "wcs_new.wcs.ctype = [\"RA---TAN\", \"DEC--TAN\"]\n",
    "\n",
    "# because FITS_rec is weird\n",
    "v_map = []\n",
    "for row in velocity_map:\n",
    "    v_map.append(np.array(row))\n",
    "\n",
    "# plotting\n",
    "fig = plt.figure(0, figsize=(8,8))\n",
    "fig.clf()\n",
    "fig.add_subplot(111, projection=wcs_new)\n",
    "plt.title(f\"SPIRE Observation {obsid}, SSW velocity map\", size=14)\n",
    "plt.imshow(v_map,  origin='lower', cmap='magma')\n",
    "plt.colorbar().set_label(label=f\"Velocity [{header['TUNIT11']}]\", size=14)\n",
    "plt.xlabel('RA', fontsize=14)\n",
    "plt.ylabel('Dec', fontsize=14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Integrated Flux\n",
    "\n",
    "When analyzing a spectrum, one is often interested in the integrated flux of spectral features. Integrated flux of spectral features is not provided in the FF products because obtaining accurate values typically requires careful analysis of the associated spectra. With this consideration, we demonstrate below a simple approach for obtaining rough estimates of the integrated flux for features detected by the FF. \n",
    "\n",
    "The proposed procedure involves fitting detected features using a Sinc model, which closely approximates the empirical line shape of the SPIRE FTS, and extracting the integrated flux analytically from the fitted line parameters. This process is similar to the method employed by the the standard spectral line fitting analysis script used by the Herschel Interactive Processing Environment (HIPE). Typically one would include the continuum in the fitted model, but in our case, we will simply use continuum subtracted spectral data where the fitted continuum is provided by the FF. \n",
    "\n",
    "The Sinc function is given by\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:sinc}\n",
    "f(\\nu) = A \\frac{\\sin(\\mathrm{arg})}{\\mathrm{arg}} \\\\\n",
    "\\mathrm{arg} = \\frac{\\pi(\\nu - C)}{W}\n",
    "\\end{equation}\n",
    "\n",
    "where $A$ and $C$ define the amplitude and centroid of the spectral feature, respectively. The width of the spectral feature, $W$, corresponds the the spectral resolution of the FTS ($\\approx$ 1.18 GHz for SPIRE). With these parameters, the integrated line flux of an unresolved spectral feature is\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:integratedFlux}\n",
    "F_{line} = \\int_{-\\infty}^{\\infty} f(\\nu) \\; d\\nu = AW\n",
    "\\end{equation}\n",
    "\n",
    "This analytic expression is used to calculate integrated line flux in the code below.\n",
    "\n",
    "Note: The Sinc function expresses decaying side-lobes which produce non-local effects in the spectrum. For this reasons, fitting a single spectral feature is not recommended. The best and most physically realistic results are obtained when simultaneously fitting all detected features in the spectrum (ie, either the entire SLW or SSW band). We will perform our analysis on the already loaded SLW spectrum from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "specIn = y_slw - cont_slw # continuum subtracted spectrum\n",
    "fitFeatures = sorted(FF_data['frequency'][slw_ind][slw_match])\n",
    "specRes = header0['SLW_HR_convol']['ACTRES']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sincModel(xdata, amp, cent, width = 1.18): \n",
    "    p2 = width/np.pi\n",
    "    arg = (xdata - cent)/p2\n",
    "    return amp*np.sin(arg)/arg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "totalModel = None\n",
    "prefs = []\n",
    "\n",
    "for i in range(len(fitFeatures)):\n",
    "    pref = f\"_{i}\"\n",
    "    prefs.append(pref)\n",
    "    freq = fitFeatures[i]\n",
    "    ind = np.where(min(abs(freq - x_slw)) == abs(freq - x_slw))[0]\n",
    "    amp = specIn[ind]\n",
    "    newMod = Model(sincModel, prefix=pref)\n",
    "    newMod.set_param_hint(pref+'amp', value=amp, vary=True)\n",
    "    newMod.set_param_hint(pref+'cent', value=freq, vary=False)\n",
    "    newMod.set_param_hint(pref+'width', value=specRes, vary=False) # set vary to True for partially resolved features\n",
    "    if totalModel == None:\n",
    "        totalModel = newMod\n",
    "    else:\n",
    "        totalModel += newMod\n",
    "        \n",
    "result = totalModel.fit(specIn, xdata=x_slw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(0, figsize=(10,6))\n",
    "plt.clf()\n",
    "plt.title(f\"Obsid {obsid}: SLW Spectrum, pixel {slw_row[0], slw_col[0]}\", fontsize=14)\n",
    "plt.plot(x_slw, specIn, linewidth=4.0, label='Data')\n",
    "plt.plot(x_slw, result.best_fit, label='Fit')\n",
    "plt.plot(x_slw, result.residual, label='Residual')\n",
    "plt.xlabel('Frequency [GHz]', fontsize=14)\n",
    "plt.ylabel('Brightness [$\\\\mathrm{W \\; m^{-2} \\; Hz^{-1} \\; sr^{-1}}$]', fontsize=14)\n",
    "plt.legend(loc=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# zoom in around a particular feature\n",
    "#\n",
    "fig = plt.figure(0, figsize=(10,6))\n",
    "plt.clf()\n",
    "plt.title(f\"Obsid {obsid}: SLW Spectrum, pixel {slw_row[0], slw_col[0]}\", fontsize=14)\n",
    "plt.plot(x_slw, specIn, linewidth=4.0, label='Data')\n",
    "plt.plot(x_slw, result.best_fit, label='Fit')\n",
    "plt.plot(x_slw, result.residual, label='Residual')\n",
    "plt.xlim((650,750))\n",
    "plt.ylim((-0.3e-16,1.1e-16))\n",
    "plt.xlabel('Frequency [GHz]', fontsize=14)\n",
    "plt.ylabel('Brightness [$\\\\mathrm{W \\; m^{-2} \\; Hz^{-1} \\; sr^{-1}}$]', fontsize=14)\n",
    "plt.legend(loc=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and print the fit results\n",
    "fitPars = result.params\n",
    "print('Frequency [GHz]     Flux [W m-2 sr-1]     Flux Error [W m-2 sr-1]\\n')\n",
    "for pref in prefs:\n",
    "    amp = fitPars[f'{pref}amp']\n",
    "    freq = fitPars[f'{pref}cent']\n",
    "    width = fitPars[f'{pref}width']\n",
    "    flux = amp.value*1e9*width.value\n",
    "    fluxErr = flux*np.sqrt((amp.stderr/amp.value)**2 + (width.stderr/width.value)**2)\n",
    "    print(f\"{freq.value:10.2f} {flux:21.3e} {fluxErr:25.3e}\\n\")\n",
    "#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above example we have assumed the spectral features are not resolved. When analyzing partially or fully resolved spectral features, best results are obtained when modeling the spectral features as Sinc convolved Gaussians. However, for a quick estimate of integrated flux, one can simply set the `vary` keyword for the `width` variable in the fitter object to `True`. The large majority of spectral features in the SPIRE FTS dataset are unresolved and the above script should be sufficient in most cases. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unit Conversion for Integrated Flux\n",
    "\n",
    "It is often convenient to express integrated flux in alternative units. Here we summarize how to convert from standard SPIRE FTS integrated line flux units to a few other commonly encountered units. For a more comprehensive overview, please see [SPIRE FTS: converting integrated line fluxes to alternative units](https://www.cosmos.esa.int/web/herschel/legacy-documentation-spire-level-2).\n",
    "\n",
    "#### Point Source Calibration\n",
    "\n",
    "Since point source calibrated spectra in SPIRE are presented in units of Janskys, employing the analytical equation for integrated line flux returns units of [Jy GHz]. To obtain integrated line flux in SI units use\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:flux_Wm2}\n",
    "F_{\\mathrm{line}} = AW \\times 10^{-17} \\qquad [\\mathrm{W \\; m^{-2}}]\n",
    "\\end{equation}\n",
    "\n",
    "where $A$ is in units of Jy and $W$ is in units of GHz. All the following unit conversions for point source calibrated spectra will be done with respect to $F_{\\mathrm{line}}$. \n",
    "\n",
    "* $[\\mathrm{W \\; m^{-2}}] \\rightarrow [\\mathrm{erg \\; cm^{-2} \\; s^{-1}}]$ \n",
    "\n",
    "This conversion requires only a multiplicative factor,\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:flux_ergcm2s}\n",
    "F' = F_{\\mathrm{line}} \\times 10^3 \\qquad [\\mathrm{erg \\; cm^{-2} \\; s^{-1}}]\n",
    "\\end{equation}\n",
    "\n",
    "* $[\\mathrm{W \\; m^{-2}}] \\rightarrow [\\mathrm{Jy \\; km \\; s^{-1}}]$\n",
    "\n",
    "This conversion requires a multiplicative factor and the rest frame emission frequency, $\\nu_0$, of the line in question using units of GHz.\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:flux_jykms1}\n",
    "F' = \\frac{F_{\\mathrm{line}} \\; (9.418 \\times 10^{19})}{\\nu_0} \\qquad [\\mathrm{Jy \\; km \\; s^{-1}}]\n",
    "\\end{equation}\n",
    "\n",
    "Alternatively, one can use the rest frame emission wavelength, $\\lambda_0$, of the line in question using units of $\\mu\\mathrm{m}$.\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:flux_jykms2}\n",
    "F' = F_{\\mathrm{line}} \\; \\lambda_0 \\; (\\pi \\times 10^{7}) \\qquad [\\mathrm{Jy \\; km \\; s^{-1}}]\n",
    "\\end{equation}\n",
    "\n",
    "* $[\\mathrm{W \\; m^{-2}}] \\rightarrow [\\mathrm{K \\; km \\; s^{-1}}]$\n",
    "\n",
    "The temperature unit Kelvins in the desired units is in reference to the so called \"brightness temperature\" and is a useful quantity for FIR/submm and radio astronomy. This conversion is somewhat more complicated in that it requires knowledge of the frequency dependent SPIRE FTS beam width. We use $\\theta_b(\\nu)$, in units of arcseconds, to denote the frequency dependent Full Width at Half Maximum (FWHM) of the beam. For SPIRE, this empirically derived curve can be obtained from the following link http://archives.esac.esa.int/hsa/legacy/ADP/PSF/SPIRE/SPIRE-S/. Both the SPIRE SLW and SSW detector arrays have their own curves. The conversion is then\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:flux_kkms}\n",
    "F' = \\frac{F_{\\mathrm{line}} \\; (1.149 \\times 10^{29})}{\\theta_b^2(\\nu) \\; \\nu_0^3} \\qquad [\\mathrm{K \\; km \\; s^{-1}}]\n",
    "\\end{equation}\n",
    "\n",
    "Note that $\\nu_0$ is again the rest frame emission frequency of the line in question in units of GHz, and $\\theta_b(\\nu)$ is the beam FWHM at the detected frequency of the line in question. Since this conversion is nontrivial, we provide a sample conversion below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq_det = 482.1 # feature detection frequency [GHz]\n",
    "freq_0 = 1000.0 # rest frame frequency for detected feature [GHz]\n",
    "amp = 1 # amplitude of detected feature [Jy]\n",
    "width = 1.18448225 # width of sinc function [GHz]\n",
    "#\n",
    "# download the FTS beam FWHM as a function of frequency\n",
    "#\n",
    "url = 'http://archives.esac.esa.int/hsa/legacy/ADP/PSF/SPIRE/SPIRE-S/fwhm_omega_SLW_HR.csv'\n",
    "beam_slw = Table.read(url,comment=\"\\s*#\",data_start=4)\n",
    "#\n",
    "freq = beam_slw['frequency']\n",
    "fwhm = beam_slw['FWHM']\n",
    "# beam FWHM at detection frequency, using linear interpolation\n",
    "theta = np.interp(freq_det, freq, fwhm)\n",
    "# calculated integrated line flux [W m-2]\n",
    "F_line = amp*width*1e-17\n",
    "# conversion to [K km s-1]\n",
    "F_ = F_line*1.149e29/theta**2/freq_0**3\n",
    "\n",
    "print(f\"Integrated line flux:\\n F_line = {F_line:.2e} [W m-2]\\n F_ = {F_:.2f} [K km s-1]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extended Source Calibration\n",
    "\n",
    "Since extended source calibrated spectra in SPIRE are presented in units of [W m-2 Hz-1 sr-1], employing the analytical equation for integrated line flux returns units of [W GHz m-2 Hz-1 sr-1]. To obtain integrated line flux in SI units use\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:flux_Wm2sr1}\n",
    "F_{\\mathrm{line}} = AW \\times 10^{9} \\qquad [\\mathrm{W \\; m^{-2} \\; sr^{-1}}]\n",
    "\\end{equation}\n",
    "\n",
    "where $A$ is the amplitude of the feature in units of [W m-2 Hz-1 sr-1], and $W$ is the width of the feature in units of GHz.\n",
    "\n",
    "*  $[\\mathrm{W \\; m^{-2} \\; sr^{-1}}] \\rightarrow [K]$\n",
    "\n",
    "The desired temperature units of Kelvins is again in relation to the so called \"brightness temperature\". For extended source calibrated spectra, the conversion requires simply a multiplicative factor and the emission rest frequency, $\\nu_0$, of the spectral feature in question in units of GHz. The conversion is given by\n",
    "\n",
    "\\begin{equation}\n",
    "\\label{eq:flux_K}\n",
    "F' = \\frac{F_{\\mathrm{line}} (1.04 \\times 10^{12})}{\\nu_0^2} \\qquad [K]\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "The Herschel/SPIRE Spectral Feature Finder is an automated tool which attempts to detect significant spectral features within SPIRE FTS data. This tool and the resulting dataset is designed to help with preliminary analysis and data mining of SPIRE spectral data and is not intended to replaced detailed analysis. In this document we have demonstrated how to retrieve FF products through a Python interface and how to query these products for features meeting specific search criteria. We also outlined simple methods for displaying FF results including plotting FF features and fitted continua over the SPIRE spectra obtained from the HSA, and displaying velocity maps for mapping observations. Finally, we provided sample code for how to extract integrated line flux for detected spectral features and outlined conversion methods for obtaining the integrated flux in other common units."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}