RCAL-965 Provide conversion from TVAC/FPS models to ScienceRawModel

[stscieisenhamer]

Resolves RCAL-965

This PR addresses the issue that TVAC/FPS data cannot be run through the pipeline, mostly due to the fact that the TVAC-related datamodels are frozen. A new method, ScienceRawModel.from_tvac_raw is introduced to do the conversion.

Tasks

• Update or add relevant roman_datamodels tests.
• Update relevant docstrings and / or docs/ page.
• Does this PR change any API used downstream? (If not, label with no-changelog-entry-needed.)
  • Write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types).
  • Start a romancal regression test (https://github.com/spacetelescope/RegressionTests/actions/workflows/romancal.yml) with this branch installed ("git+https://github.com/<fork>/rad@<branch>").

News fragment change types:

• changes/<PR#>.feature.rst: new feature
• changes/<PR#>.bugfix.rst: fixes an issue
• changes/<PR#>.doc.rst: documentation change
• changes/<PR#>.removal.rst: deprecation or removal of public API
• changes/<PR#>.misc.rst: infrastructure or miscellaneous change

[codecov]

Codecov Report

Attention: Patch coverage is 96.15385% with 4 lines in your changes missing coverage. Please review.

Project coverage is 97.26%. Comparing base (087a60d) to head (d77405e).
Report is 106 commits behind head on main.

Files with missing lines	Patch %	Lines
tests/test_models.py	93.47%	3 Missing ⚠️
src/roman_datamodels/datamodels/_utils.py	97.72%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #455      +/-   ##
==========================================
- Coverage   97.56%   97.26%   -0.30%     
==========================================
  Files          30       37       +7     
  Lines        2788     3399     +611     
==========================================
+ Hits         2720     3306     +586     
- Misses         68       93      +25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[schlafly]

Just trying to figure out how this works... if I do strun roman_elp tvac.asdf, when or where does from_tvac_raw get called?

Re schema differences, I think we should try to hit the things that romancal looks for but we don't need to copy everything over just because. The guide window xstart is an interesting case. romancal looks for it, but TVAC doesn't actually use guide windows and so if the code runs I would be fine with it without messing with guide window things.

[stscieisenhamer]

@schlafly This is all invoked in the dq_init_step. See romancal PR#1596

[stscieisenhamer]

initial regression run

[PaulHuwe]

I am a little surprised more changes aren't needed. I notice that there are regression test failures - are these expected?

[schlafly]

Thanks, I think this looks reasonable. Can you add some documentation of what exactly we're trying to accomplish here, though? Probably as a docstring in from_tvac_raw, but potentially also in the RTD if there's a reasonable place.

i.e., something like:
romancal supports processing a selection of files which use an outdated schema. It supports these with a bespoke method that converts the files to the new format when they are read in dq_init. This conversion does not do a detailed mapping between all of the new and old metadata, but instead opportunistically looks for fields with common names and assigns them. Other metadata with non-matching names is simply copied in place. This allows processing to proceed and preserves the original metadata, but the resulting files have duplicates of many entries.
If that sounds right?

I think the biggest practical issue is that a lot of the guide window stuff changed name and we'll now have those bogus fields twice, which is ugly but practical. That said, since guide windows aren't actually being used for TVAC---I think? @tddesjardins ? --- it's hard for me to justify writing code to do the conversion of all of the unused fields.

Have you reviewed the schema to see if that's the biggest area where things got renamed? I went through Tyler's L2 schema doc to try to make sure, and I think that's the case, but it would be good to have another set of eyes on it.

[stscieisenhamer]

@schlafly I've updated the docstrings in both the roman_datamodel's api and the romancal high-level description of the dq_init step.

[stscieisenhamer]

For all reviewers: I believe all the comments have been addressed. Ready for full review

[PaulHuwe]

LGTM outside of the one question I posed

[PaulHuwe]

Approved modulo test failures (likely just requiring a rebase).
I like the new function!