Exercises.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:4ca3ef763fe05d0d1e89ced119805a36a6e17a55fd6ae57542172a8faafe3981"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Unit Conversion"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Level 1\n",
      "-------\n",
      "\n",
      "What is 1 barn megaparsecs in teaspoons?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Level 2\n",
      "-------\n",
      "\n",
      "What is $3 \\mathrm{nm}^2 \\mathrm{Mpc} / \\mathrm{m}^3$ in dimensionless units?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Level 3\n",
      "-------\n",
      "\n",
      "Try and use equivalencies to find the doppler shifted wavelength of a line at $454.4\\mathrm{nm}$ if the object is moving at a velocity of $510\\mathrm{km}/\\mathrm{s}$ away from the observer."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "------------------------------"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Tables"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To start with, read in the two data files representing the master source list and observations source list.  The fields for the two tables are respectively documented in:\n",
      "\n",
      "- [master_sources](http://cxc.harvard.edu/csc/columns/master.html)\n",
      "- [obs_sources](http://cxc.harvard.edu/csc/columns/persrc.html)"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Exploring the data"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Do the following to explore the two tables:\n",
      "\n",
      "- Display the data for each table in IPython notebook using the normal way of showing the value of a variable.\n",
      "- Get a list of the column names for each table.  *Hint*: use `<TAB>` completion to easily discover all the attributes and methods, e.g. type `master_sources.` and then hit the `<TAB>` key.\n",
      "- Find the length of each table.\n",
      "- Find the column datatypes for each table."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Normally one displays a table in IPython notebook by entering the variable name in a cell and pressing `shift-Enter`.  In a terminal session the default method is using something like `print(my_table)`.  In both cases the `Table` object prefers to display only a screenful of data to prevent having a zillion lines of output if the table is huge.  If you really want to see all the data you can use the [Table.pprint](http://astropy.readthedocs.org/en/stable/api/astropy.table.table.Table.html#astropy.table.table.Table.pprint) method.\n",
      "\n",
      "- Display all the rows of the `master_sources` table using its `pprint()` method.\n",
      "- If you are working in a regular terminal window (not IPython notebook), try the `more()` method as well."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Modifying tables"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For our analysis we don't actually need the `obi` (observation interval) column in the `obs_sources` table.\n",
      "\n",
      "- Remove the `obi` column from the `obs_sources` table.\n",
      "\n",
      "The `gti_obs` column name is a bit obscure (GTI is a good time interval, FWIW).\n",
      "\n",
      "- Rename the `gti_obs` column to `obs_date`.\n",
      "\n",
      "It would be nice to have a count rate in addition to the source counts.\n",
      "\n",
      "- Add a new column `src_rate_aper_b` which is the source counts divided by observation duration in sec.\n",
      "\n",
      "Some of the sources have a negative net flux in the broad band"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Looking at the observation source data"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For each source detected in an individual observation (in the `obs_sources` table), let's look at the source flux values.\n",
      "\n",
      "- Use the matplotlib [`hist()`]( http://matplotlib.org/api/pyplot_api.html?highlight=pyplot.hist#matplotlib.pyplot.hist) function to make a histogram of the source fluxes.  Since the fluxes vary by orders of magnitude,\n",
      "  use the `numpy.log10` to put the fluxes in log space.\n",
      "  \n",
      "*HINT*: if you did not start notebook with `--pylab=inline` then try:\n",
      "```\n",
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "```\n",
      "\n",
      "- Also make the same plot but using only sources within 4 arcmin of the center.  *HINT*: use a boolean mask to select values of `theta` that are less than 4.0."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Join the master_sources and obs_sources tables"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The `master_sources` and `obs_sources` tables share a common `msid` column. What we now want is to join the master RA and Dec positions and master source names with the individual observations table.\n",
      "\n",
      "- Use the [table.join()](http://astropy.readthedocs.org/en/stable/table/operations.html#join) function to make a single table called `sources` that has the master RA, Dec, and name included for each observation source.\n",
      "\n",
      "*HINT*: the defaults for `keys` and `join_type='inner'`  are correct in this case, so the simplest possible call to `join()` will work!\n",
      "\n",
      "- *Intermediate*: Is the length of the new `sources` the same as `obs_sources`?  What happened?\n",
      "\n",
      "- *Advanced*: Make a scatter plot of the RA (x-axis) and Dec (y-axis) difference between the master source position and the observation source position.  You'll need to use `coordinates`!"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Grouped properties of `sources`"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Finally, we can look at the variability properties of sources in the CDFS using the [`group_by()`](http://astropy.readthedocs.org/en/stable/table/operations.html#id2) functionality.  \n",
      "\n",
      "This method makes a new table in which all the sources with identical master ID are next to each other.\n",
      "\n",
      "- Make a new table `g_sources` which is the `sources` table grouped by the `msid` key using the `group_by()` method.\n",
      "\n",
      "The `g_sources` table is just a regular table with all the `sources` in a particular order.  The attribute `g_sources.groups` is an object that provides access to the `msid` sub-groups.  You can access the $i^{th}$ group with `g_sources.groups[i]`.\n",
      "\n",
      "In addition the `g_sources.groups.indices` attribute is an array with the indicies of the group boundaries.\n",
      "\n",
      "- Using `np.diff()` find the number of repeat observations of each master sources.  *HINT*: use the indices, Luke.\n",
      "- Print the 50th group and note which columns are the same for all group members and which are different.  Does this make sense?  In these few observations how many different target names were provided by observers?"
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Aggregation"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The real power of grouping comes in the ability to create aggregate values for each of the groups, for instance the mean flux for each unique source.  This is done with the [`aggregate()`](http://astropy.readthedocs.org/en/stable/table/operations.html#aggregation) method, which takes a function reference as its input.  This function must take as input an array of values and return a single value.\n",
      "\n",
      "Aggregate returns a new table that has a length equal to the number of groups.\n",
      "\n",
      "- Compute the mean of all columns for each unique source (i.e. each group) using `aggregate` and the `np.mean` function.  Call this table `g_sources_mean`.\n",
      "- Notice that aggregation cannot form a mean for certain columns and these are dropped from the output.  Use the `join()` function to restore the `master_sources` information to `g_sources_mean`."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "-----------------------------------"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Handling FITS files"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Practical Exercises\n",
      "-------------------\n",
      "\n",
      "### Level 1\n",
      "\n",
      "Try and read in one of your own FITS files using ``astropy.io.fits``, and\n",
      "see if you can also plot the array values in Matplotlib. Also, examine the\n",
      "header, and try and extract individual values. You can even try and modify\n",
      "the data/header and write the data back out - but take care not to write\n",
      "over the original file!\n",
      "\n",
      "### Level 2\n",
      "\n",
      "Read in the LAT Point Source Catalog and make a scatter plot of the\n",
      "Galactic Coordinates of the sources (complete with axis labels). Bonus\n",
      "points if you can make the plot go between -180 and 180 instead of 0 and\n",
      "360 degrees. Note that the Point Source Catalog contains the Galactic\n",
      "Coordinates, so no need to convert them.\n",
      "    \n",
      "### Level 3\n",
      "\n",
      "Using Matplotlib, make an all-sky plot of the LAT Background Model in the\n",
      "Plate Carr\u00e9e projection showing the LAT Point Source Catalog overlaid with\n",
      "markers, and with the correct coordinates on the axes. You should do this\n",
      "using only ``astropy.io.fits``, Numpy, and Matplotlib (no WCS or\n",
      "coordinate conversion library). Hint: the -CAR projection is such that the\n",
      "x pixel position is proportional to longitude, and the y pixel position to\n",
      "latitude. Bonus points for a pretty colormap."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "-------------------------------------------"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Handling ASCII data tables"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* [Back to tutorial home](http://astropy4scipy2014.readthedocs.org/)\n",
      "* [Download raw notebook file](https://github.com/astropy/astropy4scipy2014/raw/master/notebooks/05_Handling_ASCII_data_tables_solutions.ipynb)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "First familiarize yourself with the available supported formats:\n",
      "\n",
      "- Open up the [Supported formats](http://astropy.readthedocs.org/en/stable/io/ascii/index.html#id1) page and click through the documentation for each format.\n",
      "- Which formats document support for table metadata like column units, comments or additional descriptions?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Answer**:\n",
      "\n",
      "The `'cds'`, `'daophot'`, and `'ipac'` formats have support for units and/or metadata.\n",
      "    "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Read the following files in the **`data/ascii/`** sub-directory and show the contents for each one.  Try to use the minimum effort and number of non-default options possible.\n",
      "\n",
      "- `'basic.dat'`\n",
      "- `'ipac.dat'`  -- from [NASA/IPAC Infrared Science Archive](http://irsa.ipac.caltech.edu/frontpage/)\n",
      "- `'daophot.dat.gz'` -- Hint: unzipping is automatic\n",
      "- `'cds/lhs2065.dat'` -- Hint: read the docs!  Note that Vizier catalog results are in CDS format.\n",
      "- `'fixed_width.dat'` -- Hint: check out the [Fixed width gallery](http://astropy.readthedocs.org/en/stable/io/ascii/fixed_width_gallery.html)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Metadata\n",
      "\n",
      "For the table from `'ipac.dat'`, there is table metadata that is in an ordered dictionary in the table `meta` attribute.  If you read the table into a variable named `dat` then you would be looking at `dat.meta`.\n",
      "\n",
      "- Print the keys of `dat.meta`.  What are they?\n",
      "- Print each of the `keywords` and their values from `dat.meta`, one per line in the format `keyword: value`.\n",
      "- For each column in the table print the column name and unit.  *HINT* remember `colnames`."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Harder reading (if you have time)\n",
      "\n",
      "Read two slightly challenging tables:\n",
      "\n",
      "- `hard1.dat` -- This is part of a real data file produced by a Sybase SQL query.\n",
      "- `hard2.dat` -- Your neighbor asked you to help computerize his garden and handed you this data file to read.  You'll need to study up on `fill_values`."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Writing\n",
      "\n",
      "Start from the table from `'ipac.dat'` and make a new table that has just first 2 lines of the first 3 columns.  This is done with:\n",
      "```python\n",
      ">>> dat3 = dat['designation', 'ra', 'dec'][:2]\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Find the output format output that will create each of the following three outputs from the table `t3`.  You can use the pattern:\n",
      "```\n",
      "dat3.write(sys.stdout, format='...', ...)  # assuming import sys beforehand\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##### Format 1\n",
      "```\n",
      "designation,ra,dec,sigra,sigdec\n",
      "J095542.55+690421.2,148.9273263,69.0725621,0.0401,0.039\n",
      "J095540.45+690439.1,148.9185759,69.0775361,0.0519,0.0494\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##### Format 2\n",
      "```\n",
      "        designation          ra        dec\n",
      "------------------- ----------- ----------\n",
      "J095542.55+690421.2 148.9273263 69.0725621\n",
      "J095540.45+690439.1 148.9185759 69.0775361\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##### Format 3\n",
      "```\n",
      "\\begin{table}\n",
      "\\begin{tabular}{ccc}\n",
      "designation & ra & dec \\\\\n",
      "J095542.55+690421.2 & 148.9273263 & 69.0725621 \\\\\n",
      "J095540.45+690439.1 & 148.9185759 & 69.0775361 \\\\\n",
      "\\end{tabular}\n",
      "\\end{table}\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##### IPAC\n",
      "\n",
      "- Now write the `t3` table back out in IPAC format.  Does it look like the output has the same units and metadata as the original?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "-------------------------------"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Celestial Coordinates"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Level 1"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Find the coordinates of the Crab Nebula in ICRS coordinates, and convert them to Galactic Coordinates."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Level 2"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Read in the sources, use the RA and Dec columns to instantiate a coordinate object, then convert to Galactic coordinates. Make a plot of latitude versus longitude."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Level 3\n",
      "-------\n",
      "\n",
      "Make an Aitoff projection map of the sources in Galactic coordinates"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}