Recommended operation of desi_tsnr_afterburner

[weaverba137]

desi_tsnr_afterburner creates the top-level exposures-${SPECPROD}.(fits|csv) files. In addition, it can create the top-level tiles-${SPECPROD}.(fits|csv) files, however, desi_tiles_completeness can also do that.

Going from fuji/guadalupe to iron to jura, this afterburner process has been run in different ways each time, and each time, effectively "by hand". This should be standardized.

More detail on jura:

1. desi_tsnr_afterburner was run on each night in jura to create an exposure summary table for each night. This was done with jura/run/scripts/afterburner/code/generate_slurm_files.sh which used tsnr_template_script.slurm (same directory).
2. The per-night exposure tables were stacked with combine_tables.py (executed inside create_merged_tables.sh).
3. create_merged_tables.sh ran desi_tiles_completeness to create the tiles summary table.
4. The summary files were created in the run directory and then copied to the top-level specprod directory.

Options for desi_tsnr_afterburner in jura:

desi_tsnr_afterburner -o ${DESI_SPECTRO_REDUX}/${SPECPROD}/run/scripts/afterburner/outfiles/exposures-${SPECPROD}-{NIGHT}.fits --prod ${SPECPROD} \
                      --aux /global/cfs/cdirs/desi/survey/observations/SV1/sv1-tiles.fits \
                      --gfa-proc-dir /global/cfs/cdirs/desi/survey/GFA/ \
                      --compute-skymags --update --nproc 64 \
                      --nights {NIGHT} &>> ${DESI_SPECTRO_REDUX}/${SPECPROD}/run/scripts/afterburner/expburner-${SPECPROD}-{NIGHT}.log

Options for desi_tiles_completeness in jura:

desi_tiles_completeness -i /global/cfs/cdirs/desi/spectro/redux/jura/run/scripts/afterburner/outfiles/exposures-jura.fits -o /global/cfs/cdirs/desi/spectro/redux/jura/run/scripts/afterburner/outfiles/tiles-jura.fits --prod jura --aux /global/cfs/cdirs/desi/survey/observations/SV1/sv1-tiles.fits
desi_tiles_completeness -i /global/cfs/cdirs/desi/spectro/redux/jura/run/scripts/afterburner/outfiles/exposures-jura.fits -o /global/cfs/cdirs/desi/spectro/redux/jura/run/scripts/afterburner/outfiles/tiles-jura.csv --prod jura --aux /global/cfs/cdirs/desi/survey/observations/SV1/sv1-tiles.fits

Recommendations:

1. It is fine to run desi_tsnr_afterburner on individual nights and then assemble them. However the scripts needed to do so should be checked in, not left for detectives to have to find deep in the ${SPECPROD}/run directory.
2. Currently desi_tiles_completeness and desi_tsnr_afterburner --tile-completeness <tile-file> do the same thing, but with different code. This is confusing and could result in changes to one script not being applied to the other. There is also a subtle difference between Table.read(), used by desi_tiles_completeness and desispec.io.read_table(), used by desi_tsnr_afterburner. These differences should be eliminated, and possibly, desi_tiles_completeness should be eliminated entirely.
3. It should not be necessary to run a script twice to create a fits file and then a csv file. The same code should generate both files, always.
4. If the summary files must be created in the ${SPECPROD}/run directory, move them to the top-level instead of leaving a copy which could cause confusion to future detectives. This step should be documented and checked in as part of the suite of afterburner scripts.

[sbailey]

For context, this is related to issue #1724 to rewrite the tsnr_afterburner for cleaner data flow and better scaling. The messy history of how it has been run differently for each prod is because the method used by each previous production didn't keep up with the scale needed for a new prod, requiring some experimentation to find what could work given the existing tools.

[sbailey]

And for completeness, desi_create_tilestable is yet another script for creating a tiles-SPECPROD.* file, completely independently of the TSNR code. That was on purpose to have something faster/lighterweight during jura while we were sorting out how to run tsnr at scale, but that does mean it is yet another place that does similar stuff. :(