JP-3243: Updates to jump detection for snowball/shower enhancements

[mwregan2]

Resolves JP-3243

This PR primarily addresses the problem of the effect of brighter-fatter in large multi-int exposures. When there are enough integrations it switches to using the numpy sigmaclip routine to find outliers.
This PR also adds in support for determining the rate of both regular Cosmic Rays and snowbals/showers and adds the rates to the Fits header.

Checklist

• added entry in CHANGES.rst (either in Bug Fixes or Changes to API)
• updated relevant tests
• updated relevant documentation
• updated relevant milestone(s)
• added relevant label(s)

[kmacdonald-stsci]

Aside from minor variable name changes, it looks good to me.

[hbushouse]

Still several questions and requests that need a response.

[hbushouse]

Latest updates look good and address the review comments. The one failing CI test is just due to the mismatch between this PR branch and jwst/master. That will be fixed once spacetelescope/jwst#7609 is merged.

Reserving final approval until the latest regtest run is finished and reviewed.

[hbushouse]

Final regtest results are at https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/765/ . All differences are due to the inclusion of the 2 new keywords and a small number of expected differences in pixel and DQ values. So I think it looks good.

[braingram]

@ddavis-stsci Is this going to impact roman?

It looks like stcal.jump.jump.detect_jump now returns 5 values:

stcal/src/stcal/jump/jump.py

Line 416 in 644392a

return gdq, pdq, total_primary_crs, number_extended_events, stddev

whereas the use in romancal expects 2 values:
https://github.com/spacetelescope/romancal/blob/7d5d5fd7e5ab86b2611b421ff42602df025120b6/romancal/jump/jump_step.py#L121

[ddavis-stsci]

Yes this change will definitely break the roman jump detection. It does look like a fairly simple fix. I'm not sure that roman will need the new parameters. Is this high priority for a release?

…

On 6/27/23 2:11 PM, Brett Graham wrote: @ddavis-stsci <https://urldefense.com/v3/__https://github.com/ddavis-stsci__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!39knE95hilXi0vRVkfydbX0TBdqpPuYl7_QVoqjFrnNAexpNyguZL_V6J9_uwiSEmvvVyodDCU8l6IwAApkr6HbT$> Is this going to impact roman? It looks like |stcal.jump.jump.detect_jump| now returns 5 values: https://github.com/spacetelescope/stcal/blob/644392ae028f436928f9efb1c5ac5f39ed1bf7a4/src/stcal/jump/jump.py#L416 <https://urldefense.com/v3/__https://github.com/spacetelescope/stcal/blob/644392ae028f436928f9efb1c5ac5f39ed1bf7a4/src/stcal/jump/jump.py*L416__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!39knE95hilXi0vRVkfydbX0TBdqpPuYl7_QVoqjFrnNAexpNyguZL_V6J9_uwiSEmvvVyodDCU8l6IwAAiKt55Mz$> whereas the use in romancal expects 2 values: https://github.com/spacetelescope/romancal/blob/7d5d5fd7e5ab86b2611b421ff42602df025120b6/romancal/jump/jump_step.py#L121 <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/blob/7d5d5fd7e5ab86b2611b421ff42602df025120b6/romancal/jump/jump_step.py*L121__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!39knE95hilXi0vRVkfydbX0TBdqpPuYl7_QVoqjFrnNAexpNyguZL_V6J9_uwiSEmvvVyodDCU8l6IwAAr4FxOTl$> — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/spacetelescope/stcal/pull/174*issuecomment-1609998580__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!39knE95hilXi0vRVkfydbX0TBdqpPuYl7_QVoqjFrnNAexpNyguZL_V6J9_uwiSEmvvVyodDCU8l6IwAAnFgKyxj$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALXCXWLSKZFY7PTFMEFTBYTXNMO5JANCNFSM6AAAAAAZIKBMUA__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!39knE95hilXi0vRVkfydbX0TBdqpPuYl7_QVoqjFrnNAexpNyguZL_V6J9_uwiSEmvvVyodDCU8l6IwAAlTOeCV5$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[hbushouse]

Is this high priority for a release?

Yes, semi-high. We want to get a jwst release out by the end of this week, and a PR we're waiting to merge in jwst depends on getting this stcal PR released as soon as possible.

[ddavis-stsci]

OK, I'll write a romancal ticket that can be merged after the stcal release. Our main regression tests will be fine until the stcal release. At that point we'll need to update for ramp fit changes in stcal as well.

…

On 6/27/23 2:33 PM, Howard Bushouse wrote: Is this high priority for a release? Yes, semi-high. We want to get a jwst release out by the end of this week, and a PR we're waiting to merge in jwst depends on getting this stcal PR released as soon as possible. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/spacetelescope/stcal/pull/174*issuecomment-1610026738__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!2EEbjW5kSNKEYNF-8ctnG_orKckSbxLr5KN9h9-g5rrmYs9xkZV_-oZnPDCl6hll1tDwMur3Hq5O18kPEd66MKYC$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALXCXWJLRLJ7CPGK4NROVCDXNMRO7ANCNFSM6AAAAAAZIKBMUA__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!2EEbjW5kSNKEYNF-8ctnG_orKckSbxLr5KN9h9-g5rrmYs9xkZV_-oZnPDCl6hll1tDwMur3Hq5O18kPEXNe4B-G$>. You are receiving this because you were mentioned.Message ID: ***@***.***>