JP-3682: split outlier detection into separate steps for each mode

.github/pull_request_template.md

@@ -84,7 +84,11 @@ This PR addresses ...
 - ``changes/<PR#>.tweakreg.rst``
 - ``changes/<PR#>.skymatch.rst``
 - ``changes/<PR#>.exp_to_source.rst``
-- ``changes/<PR#>.outlier_detection.rst``
+- ``changes/<PR#>.outlier_detection_coron.rst``
+- ``changes/<PR#>.outlier_detection_ifu.rst``
+- ``changes/<PR#>.outlier_detection_imaging.rst``
+- ``changes/<PR#>.outlier_detection_spec.rst``
+- ``changes/<PR#>.outlier_detection_tso.rst``
 - ``changes/<PR#>.tso_photometry.rst``
 - ``changes/<PR#>.stack_refs.rst``
 - ``changes/<PR#>.align_refs.rst``

changes/8722.outlier_detection_coron.rst

@@ -0,0 +1 @@
+Split outlier detection into multiple steps for different modes

changes/8722.outlier_detection_ifu.rst

@@ -0,0 +1 @@
+Split outlier detection into multiple steps for different modes

changes/8722.outlier_detection_imaging.rst

@@ -0,0 +1 @@
+Split outlier detection into multiple steps for different modes

changes/8722.outlier_detection_spec.rst

@@ -0,0 +1 @@
+Split outlier detection into multiple steps for different modes

changes/8722.outlier_detection_tso.rst

@@ -0,0 +1 @@
+Split outlier detection into multiple steps for different modes

docs/jwst/data_products/science_products.rst

@@ -354,7 +354,7 @@ source.
 Cosmic-Ray flagged data: ``crf`` and ``crfints``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Several of the stage 3 pipelines, such as :ref:`calwebb_image3 <calwebb_image3>` and
-:ref:`calwebb_spec3 <calwebb_spec3>`, include the :ref:`outlier detection <outlier_detection_step>`
+:ref:`calwebb_spec3 <calwebb_spec3>`, include an outlier detection
 step, which finds and flags outlier pixel values within calibrated images. The results of this
 process have the identical format and content as the input ``cal`` and ``calints`` products.
 The only difference is that the DQ arrays have been updated to contain CR flags. If the inputs

docs/jwst/outlier_detection_coron/arguments_coron.rst

@@ -0,0 +1,28 @@
+.. _outlier_detection_coron_step_args:
+
+Step Arguments
+==============
+
+The outlier detection step for coronagraphic data has the following
+optional arguments that control the behavior of the processing.
+For more details about step arguments (including datatypes, possible values
+and defaults) see :py:obj:`jwst.outlier_detection.OutlierDetectionCoronStep.spec`.
+
+``--save_intermediate_results``
+  Specifies whether or not to save any intermediate products created
+  during step processing.
+
+``--good_bits``
+  The DQ bit values from the input image DQ arrays
+  that should be considered 'good'. Any pixel with a DQ value not included
+  in this value (or list of values) will be ignored when resampling and flagged
+  when building the weight mask. See DQ flag :ref:`dq_parameter_specification` for details.
+
+``--snr``
+  The signal-to-noise values to use for bad pixel identification.
+  Since cosmic rays often extend across several pixels, the user
+  must specify two cut-off values for determining whether a pixel should
+  be masked: the first for detecting the primary cosmic ray, and the
+  second (typically lower threshold) for masking lower-level bad pixels
+  adjacent to those found in the first pass.  Valid values are a pair of
+  floating-point values in a single string (for example "5.0 4.0").

docs/jwst/outlier_detection_coron/index.rst

@@ -0,0 +1,13 @@
+.. _outlier_detection_coron_step:
+
+========================================
+Outlier Detection for Coronagraphic Data
+========================================
+
+.. toctree::
+   :maxdepth: 2
+
+   main.rst
+   arguments_coron.rst
+
+.. automodapi:: jwst.outlier_detection.outlier_detection_coron_step

docs/jwst/outlier_detection/outlier_detection_coron.rst → docs/jwst/outlier_detection_coron/main.rst

@@ -1,14 +1,16 @@
 .. _outlier-detection-coron:
 
-Coronagraphic Data
-==================
+Overview
+========
 
-This module serves as the interface for applying ``outlier_detection`` to coronagraphic
-image observations. A :py:class:`~jwst.datamodels.CubeModel` serves as the basic format
-for all processing performed by this step. This routine performs the following operations:
+This module serves as the interface for detecting outliers in coronagraphic
+image observations, namely those taken in the following modes:
+'MIR_LYOT', 'MIR_4QPM', 'NRC_CORON'.
+
+This routine performs the following operations:
 
 #. Extract parameter settings from input model and merge them with any user-provided values.
-   See :ref:`outlier detection arguments <outlier_detection_step_args>` for the full list
+   See :ref:`arguments <outlier_detection_coron_step_args>` for the full list
    of parameters.
 
 #. Do not attempt resampling; data are assumed to be aligned and have an identical WCS.
@@ -28,3 +30,5 @@ for all processing performed by this step. This routine performs the following o
 
 #. Update DQ arrays with flags and set SCI, ERR, and variance arrays to NaN at the location
    of identified outliers.
+
+``OutlierDetectionCoronStep`` does not use any reference files.

docs/jwst/outlier_detection_ifu/arguments_ifu.rst

@@ -0,0 +1,26 @@
+.. _outlier_detection_ifu_step_args:
+
+Step Arguments
+==============
+
+The outlier detection step for IFU data has the following
+optional arguments that control the behavior of the processing.
+For more details about step arguments (including datatypes, possible values
+and defaults) see :py:obj:`jwst.outlier_detection.OutlierDetectionIFUStep.spec`.
+
+``--save_intermediate_results``
+  Specifies whether or not to save any intermediate products created
+  during step processing.
+
+``--kernel_size``
+  The size of the kernel to use to normalize the pixel differences. The kernel size
+  must only contain odd values. Valid values are a pair of ints in a single string
+  (for example "7 7", the default).
+
+``--threshold_percent``
+  The threshold (in percent) of the normalized minimum pixel difference used to identify bad pixels.
+  Pixels with a normalized minimum difference above this percentage are flagged as outliers.
+
+``--ifu_second_check``
+  Perform a secondary check searching for outliers. This will set outliers
+  where ever the difference array of adjacent pixels is a Nan.

docs/jwst/outlier_detection_ifu/index.rst

@@ -0,0 +1,13 @@
+.. _outlier_detection_ifu_step:
+
+==============================
+Outlier Detection for IFU Data
+==============================
+
+.. toctree::
+   :maxdepth: 2
+
+   main.rst
+   arguments_ifu.rst
+
+.. automodapi:: jwst.outlier_detection.outlier_detection_ifu_step

docs/jwst/outlier_detection/outlier_detection_ifu.rst → docs/jwst/outlier_detection_ifu/main.rst

@@ -1,10 +1,13 @@
 .. _outlier-detection-ifu:
 
-Integral Field Unit (IFU) Data
-==============================
+Overview
+========
 
-This module serves as the interface for applying ``outlier_detection`` to IFU
-observations, like those taken with NIRSpec and MIRI. A :ref:`Stage 3 association <asn-level3-techspecs>`,
+This module serves as the interface for detecting outliers in IFU
+observations, namely those taken in the following modes:
+'MIR_MRS', 'NRS_IFU'.
+
+A :ref:`Stage 3 association <asn-level3-techspecs>`,
 which is loaded into a :py:class:`~jwst.datamodels.ModelContainer` object,
 serves as the basic format for all processing performed by this step.
 
@@ -22,7 +25,7 @@ by the detector PSF.
 This routine performs the following operations:
 
 #. Extract parameter settings for the input ModelContainer and merge them with any user-provided values.
-   See :ref:`outlier detection arguments <outlier_detection_step_args>` for the full list of parameters.
+   See :ref:`arguments <outlier_detection_ifu_step_args>` for the full list of parameters.
 
 #. Loop over cal files, computing nearest-neighbor differences for each pixel
    in the along-dispersion direction.
@@ -46,3 +49,5 @@ This routine performs the following operations:
 
 #. Update DQ arrays with flags and set SCI, ERR, and variance arrays to NaN at the location
    of identified outliers.
+
+``OutlierDetectionIFUStep`` does not use any reference files.

docs/jwst/outlier_detection/arguments.rst → docs/jwst/outlier_detection_imaging/arguments_imaging.rst

@@ -1,17 +1,12 @@
-.. _outlier_detection_step_args:
+.. _outlier_detection_imaging_step_args:
 
 Step Arguments
 ==============
 
-The outlier detection step has the following optional arguments
-that control the behavior of the processing.
+The outlier detection step for imaging data has the following
+optional arguments that control the behavior of the processing.
 For more details about step arguments (including datatypes, possible values
-and defaults) see :py:obj:`jwst.outlier_detection.OutlierDetectionStep.spec`.
-
-
-General Step Arguments
-----------------------
-The following arguments apply to all modes unless otherwise specified:
+and defaults) see :py:obj:`jwst.outlier_detection.OutlierDetectionImagingStep.spec`.
 
 ``--save_intermediate_results``
   Specifies whether or not to save any intermediate products created
@@ -22,7 +17,6 @@ The following arguments apply to all modes unless otherwise specified:
   that should be considered 'good'. Any pixel with a DQ value not included
   in this value (or list of values) will be ignored when resampling and flagged
   when building the weight mask. See DQ flag :ref:`dq_parameter_specification` for details.
-  Has no effect for IFU data.
 
 ``--snr``
   The signal-to-noise values to use for bad pixel identification.
@@ -32,11 +26,6 @@ The following arguments apply to all modes unless otherwise specified:
   second (typically lower threshold) for masking lower-level bad pixels
   adjacent to those found in the first pass.  Valid values are a pair of
   floating-point values in a single string (for example "5.0 4.0").
-  Has no effect for IFU data.
-
-
-Step Arguments for Imaging and Slit-like Spectroscopic data
------------------------------------------------------------
 
 ``--weight_type``
   The type of data weighting to apply to the resampled data. Available options are ``ivm``
@@ -83,37 +72,5 @@ Step Arguments for Imaging and Slit-like Spectroscopic data
   Specifies whether or not to load and create all images that are used during
   processing into memory. If ``False``, input files are loaded from disk when
   needed and all intermediate files are stored on disk, rather than in memory.
-  Has no effect for spectroscopic data. For imaging data this parameter is 
-  superseded by the pipeline-level ``in_memory`` parameter set by
-  ``calwebb_image3``.
-
-
-Step Arguments for IFU data
----------------------------
-
-``--kernel_size``
-  The size of the kernel to use to normalize the pixel differences. The kernel size
-  must only contain odd values. Valid values are a pair of ints in a single string
-  (for example "7 7", the default).
-
-``--threshold_percent``
-  The threshold (in percent) of the normalized minimum pixel difference used to identify bad pixels.
-  Pixels with a normalized minimum difference above this percentage are flagged as outliers.
-
-``--ifu_second_check``
-  Perform a secondary check searching for outliers. This will set outliers
-  where ever the difference array of adjacent pixels is a Nan.
-
-
-Step Arguments for TSO data
----------------------------
-
-``--rolling_window_width``
-  Number of integrations over which to take the median when using rolling-window
-  median for TSO observations. The default is 25. If the number of integrations
-  is less than or equal to ``rolling_window_width``, a simple median is used instead.
-
-
-Step Arguments for Coronagraphic data
--------------------------------------
-General step arguments apply to coronagraphic data. No additional arguments are used.
+  Note that this parameter is superseded by the pipeline-level ``in_memory``
+  parameter set by ``calwebb_image3``.

docs/jwst/outlier_detection_imaging/index.rst

@@ -0,0 +1,13 @@
+.. _outlier_detection_imaging_step:
+
+==================================
+Outlier Detection for Imaging Data
+==================================
+
+.. toctree::
+   :maxdepth: 2
+
+   main.rst
+   arguments_imaging.rst
+
+.. automodapi:: jwst.outlier_detection.outlier_detection_imaging_step

docs/jwst/outlier_detection/outlier_detection_imaging.rst → docs/jwst/outlier_detection_imaging/main.rst

@@ -1,21 +1,23 @@
 .. _outlier-detection-imaging:
 
-Imaging Data
-============
+Overview
+========
+
+This module serves as the interface for detecting outliers in
+imaging observations, namely those taken in the following modes:
+'FGS_IMAGE', 'MIR_IMAGE', 'NRC_IMAGE', 'NIS_IMAGE'
 
-This module serves as the interface for applying ``outlier_detection`` to direct
-image observations, like those taken with MIRI, NIRCam, and NIRISS.
 A :ref:`Stage 3 association <asn-level3-techspecs>`,
 which is loaded into a :py:class:`~jwst.datamodels.ModelLibrary` object,
 serves as the basic format for all processing performed by this step.
 This routine performs the following operations:
 
 #. Extract parameter settings for the input models and merge them with any user-provided values.
-   See :ref:`outlier detection arguments <outlier_detection_step_args>` for the full list of parameters.
+   See :ref:`arguments <outlier_detection_imaging_step_args>` for the full list of parameters.
 
 #. By default, resample all input images to the same output WCS. The resample process is
    controlled by the ``fillval``, ``pixfrac``, ``kernel``, and ``good_bits`` parameters;
-   see the :ref:`outlier detection arguments <outlier_detection_step_args>` for more information.
+   see the :ref:`arguments <outlier_detection_imaging_step_args>` for more information.
    Resampling can be turned off with the ``resample_data`` parameter.
 
    * Compute an output WCS that is large enough to encompass all the input images.
@@ -125,3 +127,5 @@ is input to the step, the memory behavior of the step is read from the ``on_disk
 of the ModelLibrary object, and the ``in_memory`` parameter of the step is ignored.
 When running ``calwebb_image3``, the ``in_memory`` flag should be set at the pipeline level,
 e.g., ``strun calwebb_image3 asn.json --in-memory=False``; the step-specific flag will be ignored.
+
+``OutlierDetectionImagingStep`` does not use any reference files.