JP-3364 allow wcs shifts for IFU individual exposures in cube_build

[jemorrison]

Resolves JP-3364

Closes #7876

This PR addresses allows the user to provide a list of ra and decs for each input file. The shifts are provided in arc seconds
prior to the projections into the xi/eta space (tangent plane). This allows for a cube-space fix to the known WCS issue based on user-defined inputs.  For each input file in the association cube_build creates the cubes there must be a ra and dec shift provided in an OFFSET file.

An ifuoffset.schema.yaml file was added to the cube_build step (instead of stdatamodels). This schema file defines the format of the offset file.
The offsets are defined by a filename, raoffset, and decoffset list. The units of the offset are required to be arcsec and are required to be in the offset file. This offset file must be an asdf file.
An example of how to make the offset files is provided in the documentation.

Checklist for PR authors (skip items if you don't have permissions or they are not applicable)

• added entry in CHANGES.rst within the relevant release section
• updated or added relevant tests
• updated relevant documentation
• added relevant milestone
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below.
  How to run regression tests on a PR
  https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1723/
• All comments are resolved
• Make sure the JIRA ticket is resolved properly

[jemorrison]

Example of how to make an offset file:
It is assumed the user will determine the ra and dec offsets for each file
filename = list of filenames that are the same as in the association
raoffset = list of ra shifts cooresponding to each filename
decoffset = list of dec shifts cooresponding to each filename
import asdf
offsets = {}

offsets['filename'] = []
offsets['raoffset'] = []
offsets['decoffset'] = []
for i in range(num):
offsets['filename'].append(filename[I])
offsets['raoffset'].append(ra_center1[i])
offsets['decoffset'].append(dec_center1[i])

af = asdf.AsdfFile({'offsets':offsets})
af.write_to(input_dir + 'offsets.asdf')

[codecov]

Codecov Report

Attention: Patch coverage is 73.11828% with 25 lines in your changes missing coverage. Please review.

Project coverage is 61.81%. Comparing base (dc2ce65) to head (7683808).
Report is 40 commits behind head on master.

Files with missing lines	Patch %	Lines
jwst/cube_build/ifu_cube.py	63.63%	20 Missing ⚠️
jwst/cube_build/cube_build_step.py	86.84%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8720      +/-   ##
==========================================
+ Coverage   61.78%   61.81%   +0.02%     
==========================================
  Files         377      377              
  Lines       38836    38915      +79     
==========================================
+ Hits        23996    24056      +60     
- Misses      14840    14859      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[karllark]

The code looks good to me.

I did notice there was no update to the docs. This is important as the format of the file and the units of the offsets are needed info for users. These were questions I had that I found by reviewing the code changes. Updating the docs with this new option should be done.

[jemorrison]

@karllark I will definitely update the docs. I just want to make sure the approach I have made is what is expected. I need to include docs on how to make the asdf offset file.

[jemorrison]

@tapastro I have included in a comment the code I used to create the asdf offset file. Does look about right ?
The code is expected a certain format. I don't know of any way to make it general so I was thinking we could provide an example of how to make the ra,dec shifts needed in the offset file

[karllark]

@karllark I will definitely update the docs. I just want to make sure the approach I have made is what is expected. I need to include docs on how to make the asdf offset file.

Great. Yes, the approach makes sense to me.

[drlaw1558]

There was a comment about how you made the asdf file? I think I'm making mine incorrectly, as I'm getting errors using it.

[jemorrison]

@drlaw1558
Example of how to make an offset file:
It is assumed the user will determine the ra and dec offsets for each file
filename = list of filenames that are the same as in the association
raoffset = list of ra shifts cooresponding to each filename
decoffset = list of dec shifts cooresponding to each filename
import asdf
offsets = {}

offsets['filename'] = []
offsets['raoffset'] = []
offsets['decoffset'] = []
for i in range(num):
offsets['filename'].append(filename[I])
offsets['raoffset'].append(ra_center1[i])
offsets['decoffset'].append(dec_center1[i])

af = asdf.AsdfFile({'offsets':offsets})
af.write_to(input_dir + 'offsets.asdf')

[drlaw1558]

One thing I noticed- the code matches on local filenames, while associations can sometimes be full-path filenames. That's probably ok, but worth noting in the documentation that the ASDF file should contain only the filenames and not the full paths otherwise it won't match them properly.

We'll also definitely need to give the above code to create input files explicitly in the documentation as ASDF files are very non-intuitive for most folks.

Another thought: we probably want the offsets to be used in actual arcseconds rather than difference in sky coordinates. I.e., in the test case I'm using (PID 1523) I'm specifying an offset of 1.0 in both RA and in DEC, but the source is barely moving in the RA direction because of the cos(dec) term (this case is at 70 degrees South). We could choose either convention, but I think having offsets be consistent units of arcsec regardless of where we are on the sky makes most sense, and handle the cos(dec) translation to sky coordinates under the hood.

[jemorrison]

@drlaw1558 I was wondering about the cos(dec) term. Since the user is creating the offset file I was not sure what made the most sense in how to specific what the ra offset should be. I used IFU cubes created from a single file and then centroided on the source to figure out a set of ra and dec offsets than I then feed back to cube build. I assume the cos(dec) term we need to apply is the dec of CRVAL2. Is this too circular ? Should we provide one work flow on how to do this. 1. Build a cube at a given crval1 and crval2 determine the ra and dec offset. This offset file is then provided to a new run of cube_build with the same crval1, crval2. Or am I thinking about this too narrowly

[drlaw1558]

@jemorrison These cubes are so small that the dec we use should be irrelevant; CRVAL2 should be fine, it doesn't matter if the new cubes use slightly different CRVAL1/CRVAL2 before/after the pointing tweak. Given that offsets are generally sub-arcsecond, I'd expect that any inconsistencies in which dec we use for the cos(dec) term should only matter at the few nano-arcsec level.

We could also use the nanmedian of the input declinations, or the header keyword pointing information, whichever is simplest.

[jemorrison]

I don't think I have the cos(dec) term correct. I am dividing by it - thinking when you are at large dec you want the offset to larger in ra . But the more I think about this I think I should just have this reviewed and someone can point out my mistakes cause that does not feel right to me. I might need a clearer description in the docs on that ra and dec offset - how they are determined - absolute not relative.

[jemorrison]

I can not get the readthedocs built. Locally I get failures I have not idea how to fix:
(Extension error (sphinx_automodapi.automodsumm):
Handler <function process_automodsumm_generation at 0x12c99e840> for event 'builder-inited' threw an exception (exception: No module named 'jwst.resample.resample_utils')

In the past I have used the cl results to look for doc error but I am not finding what is in the 'Details' at all helpful

[melanieclarke]

I can not get the readthedocs built.

I tested your branch locally in my environment, and was able to build them (with some warnings), so it's probably an issue with your setup/local repo rather than the changes to the docs. Are you running make clean before make html?

[melanieclarke]

Also, searching for 'warn' in the details from the docs build on CI, I think this is the relevant problem:

/home/docs/checkouts/readthedocs.org/user_builds/jwst-pipeline/checkouts/8720/docs/jwst/cube_build/arguments.rst:134: WARNING: Explicit markup ends without a blank line; unexpected unindent.
/home/docs/checkouts/readthedocs.org/user_builds/jwst-pipeline/checkouts/8720/docs/jwst/cube_build/arguments.rst:153: ERROR: Unexpected indentation.

[melanieclarke]

And one more comment, about getting the cosine term right -- it might be better to use an astropy SkyCoord or similar to do the offset math for you.

[jemorrison]

@melanieclarke what version of sphinx should I have. I have Sphinx v7.4.7

[melanieclarke]

@melanieclarke what version of sphinx should I have. I have Sphinx v7.4.7

The Sphinx version shouldn't be terribly important, as long as it's <8. Upgrading mine to 7.4.7 I am still able to build the docs on your branch.

From the error message, it sounds like it's looking for a jwst import it can't find. If you're able to run the pipeline okay from the environment you're making the docs in, then it may be an issue with some old documentation artifacts or something like that - you could try doing a git clean in the docs directory to remove.

Otherwise, I'd recommend rebuilding your python environment and see if a fresh start sorts you out.

[drlaw1558]

Looks correct to me; inputs are assumed to be offsets in units of arcsec. Conversion to actual change in values of the RA coordinate divides this by cos(dec), which will result in making the value difference larger near the poles. Empirically, the shift that I see in data cubes processed with the code now look as expected too- specifying an offset of (1,1) moved the source the same number of spaxels N and E in the data cube.

[jemorrison]

@drlaw1558 @nden
I have made a draft IFUOffsetModel and I am trying to build an offset file with it. I have never done this before so I could use some pointers. I made it an ReferenceFileModel because I did not see another Class model I could use.
But that gives a bunch of warnings of not having meta data.
I really did not want this to be difficult to the user to create so I set the values that it was complaining about
pedigree, author, use after, description. I tried to also set 'retype' and 'instrument.name' - but it is not accepting my values

Is this the right approach ?
spacetelescope/stdatamodels#322

[melanieclarke]

One last regtest run to make sure there are no interactions with other merged PRs:
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1739/

[melanieclarke]

No relevant regtest failures. Merging now!