Support reading zbest redshift files

[weaverba137]

Some early tiles in the daily specprod have zbest instead of redrock files. To support loading daily into a database, these need to be read by desispec.scripts.zcatalog. In this PR:

• read_redrock detects whether a zbest file is being opened and attempts to reconstruct its output as best as possible to match a redrock file.
• Do not require the tiles file to be a FITS file.
• When writing out zcatalog files, include comments on columns as well as units.

Open questions:

• Do we want to put any more effort toward recovering missing data? For example, MEAN_PSF_TO_FIBER_SPECFLUX in the (coadded) fibermap table or the entire TSNR2 table missing from zbest files?
• In order to reduce code complexity, in this PR, read_redrock always converts the per-exposure fibermap into a astropy.table.Table. Previously, the per-exposure fibermap would only be a Table if recoadd_fibermap=True. We need to determine whether this introduces a significant performance overhead.
• While testing, I noticed the output zcatalog files did not have comments for every column. I thought I reported this elsewhere, but I can't find where. Does anyone know? The missing columns were:
  • FIRSTNIGHT
  • NUMEXP
  • NUMTILE
  • MIN_MJD
  • MAX_MJD
  • MEAN_MJD
  • IN_COADD_(B|R|Z), in the EXP_FIBERMAP HDU.

Some additional operational testing should probably be done before merging.

[coveralls]

coverage: 30.128% (-0.04%) from 30.164%
when pulling 46c167a on read-zbest
into 97b1748 on main.

[akremin]

Thanks, Ben. I didn't sift through every line close enough to understand every detail, but the changes look good. I ran a few tests under this branch that passed. All tests seem to work as expected.

I ran the following:

read_redrock('/global/cfs/cdirs/desi/spectro/redux/daily/tiles/cumulative/2184/20210516/zbest-0-2184-thru20210516.fits', group='cumulative', recoadd_fibermap=True, pertile=True)

read_redrock('/global/cfs/cdirs/desi/spectro/redux/daily/tiles/cumulative/2184/20210516/zbest-6-2184-thru20210516.fits', group='cumulative', recoadd_fibermap=True, pertile=True)

read_redrock('/global/cfs/cdirs/desi/spectro/redux/daily/tiles/cumulative/6198/20240409/redrock-0-6198-thru20240409.fits', group='cumulative', recoadd_fibermap=True, pertile=True)

and

read_redrock('/global/cfs/cdirs/desi/spectro/redux/daily/tiles/cumulative/6198/20240409/redrock-6-6198-thru20240409.fits', group='cumulative', recoadd_fibermap=True, pertile=True).

[weaverba137]

At the data telecon on 2024-09-24, there appeared to be a consensus that we should not put additional effort toward recovering data missing from zbest files. That leaves two open questions: does a particular Table() conversion impact performance, and new/unknown columns.

The second open question is also mentioned in desihub/desidatamodel#199.

[sbailey]

I verified with runs of this branch and main, that the switch from numpy structured arrays to astropy Table has negligible impact on the overall runtime for desi_zcatalog and that they produce identical catalogs, i.e. nothing funny with bytes vs. strings, masked values, etc.