Switch resample to the new drizzle package API

[mcara]

Resolves JP-3685
Resolves JP-3713

Closes #8638
Closes #8721

This PR switches resample step to the new drizzle API proposed in spacetelescope/drizzle#134

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.fits_generator.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[codecov]

Codecov Report

Attention: Patch coverage is 94.55446% with 11 lines in your changes missing coverage. Please review.

Project coverage is 61.93%. Comparing base (c751b4e) to head (085265e).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/resample/resample.py	93.75%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8866      +/-   ##
==========================================
- Coverage   61.96%   61.93%   -0.04%     
==========================================
  Files         377      376       -1     
  Lines       38765    38675      -90     
==========================================
- Hits        24022    23954      -68     
+ Misses      14743    14721      -22     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[emolter]

Thanks Mihai, looks good overall and my comments are minor. It needs a changelog entry. It would also be good to run our regression test suite; do you have permissions and know how to do that?

[mcara]

Regression tests running here: https://github.com/spacetelescope/RegressionTests/actions/runs/11247877551

[nden]

Is a deepcopy necessary here?

[mcara]

I am not sure. It felt to be a safer way because this method sets array_shape property of the WCS object. If there is no drawback in modifying inputs, deepcopy can be omitted.

[nden]

What happens here if shape is None ?

[braingram]

Is the new drizzle API intended to make direct calls to tdriz tblot unnecessary?

[mcara]

Is the new drizzle API intended to make direct calls to tdriz tblot unnecessary?

In general, new drizzle API is not "intended" to rely on tdriz and such but these calls are available to the developers. That is, the Python API was the one that was modified, not the C-based function API.

The purpose of this PR is to adapt the current code that otherwise would break to the new API when needed. We know that we are going to switch to the new API anyway via other PRs in jwst and stcal when the resample code is migrated there. So this PR simply switches to the new drizzle code with minimal effort in order to understand and OKify (when reasonable) regression test changes due to (minor) bug fixes and the Python class redesign.

CC: @nden

[emolter]

This looks mostly good to me now, but there are 12 failing tests in the regression test run and only 5 failing tests in last night's nightly run. Do these changes look to you like they are caused by this PR, and are any of them large?

[nden]

@mcara I may be confused about what this PR is meant to accomplish.

• How is it using the new API in drizzle? The only change here is calling calc_pixmap from drizzle with a modified signature. The rest of the changes are simply reformatting lines (I prefer these not be included) or fixing white spaces.
• Changing the signature of calc_pixmap means the input WCS needs to be modified to include the change in shape, which means a deepcopy of the WCS is needed. Deep copies of WCS objects are usually expensive and I would prefer the old signature in order to avoid the copies.
• The new implementation of calc_pixmap computes the input pixels in a different way and does not follow the adopted convention (using grid_from_bounding_box) . This means the inputs are probably slightly off the center of pixels and this will lead to different results. Is there a reason not to use the convention? If you are concerned that gwcs will become a dependency of drizzle then the function should go in stcal (there might even be one there already).

[mcara]

After the first commit, regression tests are passing, see https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1787

[mcara]

After commit 91805f7, regression tests were started here: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1793/

[mcara]

Regression tests after complete switch to the new Drizzle class (commit d323fee): https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1797/

There are 4 failing tests. Failures are unrelated.

[mcara]

Regression tests after complete switch to the new Drizzle class + performance improvements (commit 0858ed5): https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1798/

Results: same 4 unrelated failures. Regression tests appear to run shorter.

[mcara]

Also, romancal regression tests do not conflict with the new drizzle - see https://github.com/spacetelescope/RegressionTests/actions/runs/11410576103/job/31764795945

I spoke to @schlafly and he did not object to making a release of the drizzle package.

[mcara]

One more thing: before merging, would it be OK to remove resample_variance_arrays() that's no longer used - see

jwst/jwst/resample/resample.py

Lines 704 to 749 in 90ba645

	def resample_variance_arrays(self, output_model, input_models):
	"""Resample variance arrays from input_models to the output_model.
	Variance images from each input model are resampled individually and
	added to a weighted sum. If ``weight_type`` is 'ivm', the inverse of the
	resampled read noise variance is used as the weight for all the variance
	components. If ``weight_type`` is 'exptime', the exposure time is used.
	The output_model is modified in place.
	"""
	self._init_variance_arrays()
	output_shape = self.output_array_shape
	log.info("Resampling variance components")
	with input_models:
	for i, model in enumerate(input_models):
	# compute pixmap and inwht arrays common to all variance arrays:
	inwht = resample_utils.build_driz_weight(
	model,
	weight_type=self.weight_type, # weights match science
	good_bits=self.good_bits,
	)
	pixmap = resample_utils.calc_gwcs_pixmap(
	model.meta.wcs,
	output_model.meta.wcs,
	model.data.shape,
	)
	in_image_limits = resample_utils._resample_range(
	model.data.shape,
	model.meta.wcs.bounding_box,
	)
	iscale = self._iscales[i]
	self._resample_variance_arrays(
	model=model,
	iscale=iscale,
	inwht=inwht,
	pixmap=pixmap,
	in_image_limits=in_image_limits,
	output_shape=output_shape,
	)
	del inwht
	input_models.shelve(model, i)
	# We now have a sum of the weighted resampled variances.
	self._compute_resample_variance_totals(output_model)

and self._iscales that's needed only to support resample_variance_arrays() (for backward compatibility - I highly doubt that anyone would be calling this function at all by itself)?

[emolter]

If it's no longer used, yes, I think you can and should remove it

[melanieclarke]

One more thing: before merging, would it be OK to remove resample_variance_arrays() that's no longer used

I agree with Ned - that function should be removed since it's no longer used. It was just a helper function to compute the variance arrays while resampling. Since the relevant calculations have been moved inside the loop over models instead of doing them after the loop has completed, there's no reason to keep the old function.

[mcara]

Regression tests after the last two commits (pin drizzle + remove _iscales and resample_variance_arrays()): https://github.com/spacetelescope/RegressionTests/actions/runs/11494300632

No failures.