Image Quality for 90Prime Operations

[dsand]

What do we want out of a 90Prime reduction pipeline associated with operations? what do I mean by this?

I mean a pipeline that is capable of keeping an eye on data quality and conditions in near real time to find problems early and to catalog conditions on the mountain. Ideally it would monitor:

1. Noise level in bias frames.
2. Counts in flat frames.
3. Image quality (i.e. seeing) across the focal plane. Be aware when focus has changed significantly.
4. Sky background.
5. Rough image depth.

I'm sure there is more that I'm not thinking of.

[davner]

Would people be interested in sky brightness? I would find it useful for monitoring conditions on site.

It would require source extraction using something like photutils or sextractor. Then astrometric image registration using some code like SCAMP and pass those stars to a catalog to find the zeropoint of the field (theres another piece of information we can supply). Then use this basic equation to get a rough measurement.

nsb = zeropoint - 2.5 * log10(( backgroun/exposure time) / platescale ^2)

I would like to see this being general enough to be used on all imagers with just a config.py file with logging of this information, an api to call with json information returned. I could see this being used for many instruments (new and old) at different sites. I would be interested in using my non-work time helping develop parts of this.

[dsand]

Yes, I agree. I sorta meant this by #4 in my initial note. Also sounds good that we should make this a general tool for the imagers. This sounds like a good project for someone.

One other thing to add:
5. Pointing accuracy. Or something similar. Using scamp.

[lund0946]

I see this as two separate pieces of software.

A calibration monitor that keeps track of biases and flats. It monitors the noise level in the biases, the counts in the flats, and also the number that are taken in each binning mode or binning/filter combination for flats.

Then a nighttime data quality assessment monitor that measures the image quality, sky background, pointing offset, and image depth.

[dsand]

I don't have strong feelings about whether this should be one or two pieces of software, as long as the 5 or 6 metrics mentioned above are databased somehow.

[davner]

Yea, it could just be one package that does different operations for different IMAGETYPEs. @dsand any new postdocs/grads that would want to take lead on this?

[lund0946]

I can work on this a bit while I'm up on the next ARTN run focusing on the functions for measuring everything.

• We start with a very basic reduction pointing to the science image, biases, and flats. This can record bias noise levels and flat counts.
• Then make function to get the astrometric solution for the image and save the pointing offset information.
• Then call SourceExtractor or photutils.
• Then load a reference catalog.
• Then measure the IQ using the FWHM and correcting for airmass.
• Then we can measure the image depth and using the reference catalog calculate an extinction due to cloud cover.
• Then measure the BG of the image using some nominal zeropoint.
• Then jsonify these values for the webpage.

[dsand]

I'm assigning @lund0946 to this. And reminding him he should start a github repo with what he has so far.

[dsand]

I think I'm going to re-assign Griffin to this issue. He is going to write the image quality pipeline, and he has some aspects of it working. A decent description of the IQ pipeline is in the Operations Plan:
https://docs.google.com/document/d/1OZaBQWCWEBuuUef1vsO267xQi8jbTqFPYP6kNZargxA/edit?usp=sharing

[dsand]

Maybe @griffin-h can update this at some point. OR, close it and start new more relevant issues for the BANZAI-based pipeline?

[griffin-h]

I made a lot of progress installing and customizing BANZAI and its associated services during the last engineering run. I left installation notes in the engineering run log. I didn't get the astrometry.net indices and ATLAS photometric catalog until late in the run, so it hasn't been tested end to end yet, but it should be ready for testing next time. The outputs should be mosaicked images that are astrometrically and photometrically calibrated, with source catalogs as a separate FITS extension.

Action items for the February run, roughly in order of priority:

• Test the existing pipeline end to end
• Automate the installation a little more (custom Dockerfile and maybe docker-compose)
• Get the BANZAI logs into a database and/or grafana. This might include breaking out FWHM as a function of position, from the source catalogs.
• Investigate speeding up the cosmic ray step
• Longer-term: customizing BANZAI tests to 90Prime needs (e.g., header keywords)

I'm also working on a second list of header suggestions for @mplesser that will help with BANZAI, but I want to make sure everything works before asking him to make more changes.

[dsand]

@griffin-h continues to make strong progress on the image quality pipeline. It now runs on incoming 90Prime data. Griffin is working on optimizing it a little bit, and to plot up data quality metrics in near real time.