Exposure time header possible metadata errors for NIRSpec IFU data level 3 products

[stscijgbot-jp]

Issue JP-3752 was created on JIRA by Marshall Perrin:

Various header metadata values related to exposure times seem to be problematic for at least some NIRSpec IFU observations. I suspect this may be related to summing the exposure times for the NRS1 and NRS2 detector observations, taken simultaneously, which is a behavior that is likely not as intended (and is likely to be surprising to users).  This may be a more generalized issue affecting other modes as well, but I noticed it first with NIRSpec data.

I base this bug report in part on the [summary of header keywords available in JDox|[https://jwst-docs.stsci.edu/accessing-jwst-data/jwst-science-data-overview/jwst-time-definitions]], and comparison with expected metadata from program plans and APT files.

Examples showing a TELAPSE value 2x higher than expected:

Program 3399 obs 1. Filename jw03399-o001_t001_nirspec_g395h-f290lp_s3d.fits on MAST.

• APT reports Total Exposure Time = 4901.8 s.
• Pipeline output headers are all about 2x higher: 
  XPOSURE=             9336.896 / [s] Effective exposure time                    
  TMEASURE=    8870.045119999999 / [s] Measurement time                           
  TELAPSE =             9803.744 / [s] Total elapsed exposure time                
        

Program 6362 obs 1. Filename jw06362-o001_t001_nirspec_g395h-f290lp_s3d.fits on MAST.

• APT reports Total Exposure Time = 11817.0 s
• Pipeline output header are all higher, by various amounts up to 2x: 
  XPOSURE=   18907.200000000004 / [s] Effective exposure time                    
  TMEASURE=   14180.401079999998 / [s] Measurement time                           
  TELAPSE=              23634.0 / [s] Total duration of exposure         

The TELAPSE values are precisely 2x greater than APT's total exposure time.

I think we may need to clarify and/or improve documentation of the desired behavior of those keywords for level 3 data products. For combined multiple exposures using detectors exposed simultaneously (e.g. NRS1 and NRS2) I'm not sure it's the correct desired behavior to sum simultaneous exposures into the calculation of XPOSURE, TMEASURE, and TELAPSE?  That double counts exposure times in a way inconsistent with APT and ETC behaviors.  

A related potential doc inconsistency is the JDox page linked above states that TELAPSE should equal the difference between MJD-BEG and MJD-END. For 6362:1 the difference of those header keyword values is 12473 s, or nearly a factor of 2 less than the reported TELAPSE value.  Simple differences of MJD times are not going to yield correct values for level-3 combined products.  It may be worth updating that JDox page to clearly state how these keywords work both for individual L1/L2 data products and also for L3 combined data products. 

[stscijgbot-jp]

Comment by Marshall Perrin on JIRA:

FYI Bryan Holler I would be interested to hear your take on this, based on your tech report about exposure time metadata and the JDox page you pointed me towards. 

[stscijgbot-jp]

Comment by Bryan Holler on JIRA:

Marshall Perrin Is this related to the conversation we had a while back with your postdoc? If so, I agree that the pipeline should not report the sum of the times from the NRS1 and NRS2 detectors in the Level 3 file. This makes sense for mosaics, but not for detectors that are illuminated simultaneously. I wonder if this is also an issue for MRS observations, or NIRCam imaging observations using all 4 SW detectors?

But to explicitly answer your question, I think this is more related to instrument operations with multiple detectors than to the timing memo. Based on our previous conversation, the values reported would be correct if not for the "double counting" for multiple detectors.

[stscijgbot-jp]

Comment by Marshall Perrin on JIRA:

Bryan Holler -  Yes, this related to that conversation a while back, which it slipped through the cracks for me file a ticket for at that time, but I just rediscovered the issue myself again, and confirmed it's still present in 1.15.1 now in DMS operations. 

Specific question for you. I was wondering if, during the course of preparing that timing memo, did any discussion ever come up about Level 3 data products?  I don't see that mentioned in the report, understood, but I'm wondering if it was discussed at all about how these keywords are handled for combined products.

[stscijgbot-jp]

Comment by David Law on JIRA:

Looks like we did consider this issue a few years ago, in JP-2263

At the time, for consistency with imaging mosaics we opted to leave the double-counting as is.  I.e., in an imaging mosaic the total time reported is the total effective time across the entire area, even though any one tile in the mosaic won't be that deep.  The spectral case is somewhat similar, just with a mosaic in the wavelength domain rather than the spatial domain.  This gets even weirder for the MRS, where some spectral segments are taken at the same time on the same detector, some at the same time on different detectors, and some at different times.  What does total integration time mean in such a case?  Total integration time at a given wavelength, ignoring band overlaps?  If the IFU observations are mosaiced too, is that total integration time at a given wavelength over the entire FOV or at a given location in the FOV too?

[stscijgbot-jp]

Comment by Bryan Holler on JIRA:

Marshall Perrin The first version of the timing memo was led by Jeff Valenti and I inherited it for the last 2 revisions. In my time as owner of the document, there was never any discussion about pipeline processing steps, the focus was always on detector behavior, resets, overheads, etc. So, implicitly, we were considering only detector-level processing (i.e., up to Level 2b).

David Law You make a good point about how times are reported for Level 3 products. In a spectral region of overlap, or overlap between mosaic tiles, I would consider the time to be doubled, but outside those overlap regions the time shouldn't be doubled. It really sounds like a human brain needs to be in the loop to interpret the meaning of the times in the combined products. Maybe some additional information needs to be added to the JWST Time Definitions page to provide caveats for the combined products?