rectification of spectral traces

[oczoske]

Convenience method to produce rectified 2D spectra (wavelength, position along slit) from a detector readout.

• base method is SpectralTrace.rectify(hdulist, ...)
• user interface (e.g. for MICADO and METIS LSS) is SpectralTraceList.rectify_traces(hdulist)

The main input is hdulist, the product of OpticalTrain.readout(). The output is a HDUList where each extension correspond to the rectified spectrum from one trace in the trace list (only traces within the wavelength range of the filter are considered).
For now, it is mandatory to provide the slit length by the keywords xi_min and xi_max (e.g. -4, 4 for METIS LSS, -1.5, 1.5 for the MICADO short slit) explicitely on the command line. Alternatively, these values can be added manually to the header of hdulist[0] as keywords HIERARCH INS SLIT XIMIN and ... XIMAX . This is stupid but I could not find a way to get these values automatically...

[codecov]

Codecov Report

Patch coverage: 21.05% and project coverage change: -0.48 ⚠️

Comparison is base (79d0aea) 75.41% compared to head (f1a8239) 74.93%.

Additional details and impacted files

@@              Coverage Diff               @@
##           dev_master     #227      +/-   ##
==============================================
- Coverage       75.41%   74.93%   -0.48%     
==============================================
  Files             149      149              
  Lines           15283    15406     +123     
==============================================
+ Hits            11525    11544      +19     
- Misses           3758     3862     +104     

Impacted Files	Coverage Δ
scopesim/effects/spectral_trace_list.py	52.69% <12.50%> (-12.94%)	⬇️
scopesim/effects/spectral_trace_list_utils.py	62.44% <17.64%> (-11.31%)	⬇️
...esim/tests/tests_effects/test_SpectralTraceList.py	96.07% <100.00%> (+0.72%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[oczoske]

Here's a notebook (hidden in a zip so that github accepts it) that demonstrates the rectification for METIS (simulating MICADO is a bit too heavy for a notebook).
demo_rectify_traces.zip

[hugobuddel]

Suggested change

	for hdu in hdulist:
	for ihdu, hdu in enumerate(hdulist):

[oczoske]

That suggestion does not work like this - the reason is that ihdu is used to index the interps list, which is not exactly parallel to hdulist (the primary du of hdulist is jumped over). enumerate(hdulist[1:]) should work for all existing cases, but I wouldn't want to rely on that.
I had to rerun the notebook to realise this, which is good justification for including it in automatic testing.

[teutoburg]

Suggested change

	for hdu in hdulist:
	imghdulist = [hdu for hdu in hdulist if isinstance(hdu, fits.ImageHDU)]
	assert len(imhdulist) == len(interps)
	# from Python 3.10 on, the above assert statement could be replaced by passing
	# strict=True to the zip below
	for hdu, interp in zip(imghdulist, interps):

As far as I can tell, the individual hdus are not modified in this loop, only accessed. Only drawback I could think of is memory consumption, because we essentially create a copy of hdulist. Either way, I don't want to overcomplicate things, and the original implementation is probably just fine.

[hugobuddel]

Summary of our (@oczoske, @hugobuddel, @teutoburg) in-person discussion earlier today:

• The rectification isn't directly relevant for ScopeSim itself, however this was popular functionality from SpecCado, and at the moment fairly tied to ScopeSim, so fine to include it.
• The testing is currently too minimal, the plan to improve it is
  • Soon: Add the notebook referenced above to the IRDB, so at least this code is regularly ran, and somewhat documented.
  • Later: convert the notebook to a test (and or notebook) that uses one of the mock packages in ScopeSim itself. This would add coverage.
• The code is currently (by necessity) tied too much to ScopeSim and to METIS/MICADO specifics. In theory, the headers of the produced FITS file should contain all the information required to perform the rectification. E.g. it should not be necessary to access from_currsys. This code could perhaps be used as a basis to verify that the headers are indeed complete. Some information that needs to be recoverable from the headers:
  • the tracelist
  • the width or length of the slit
  • the wavelength range
  • the filter
• Finally, there are some minor stylistic things that perhaps should be improved, but nothing that would stop getting this merged:
  • the code relies too often on kwargs where this is not necessary
  • enumerate should be used

We can merge this as far as I'm concerned, but not sure whether @oczoske still wants to add some things, so I'll just approve.

[hugobuddel]

I'll merge this, because the corresponding PR in the IRDB has been merged, and now the CI there is failing.