Merge pull request #795 from dirac-institute/no_stamp_filter

Remove C++ Stamp Filtering

Author: jeremykubica

docs/source/user_manual/results_filtering.rst

@@ -21,29 +21,6 @@ Relevant light curve filtering parameters include:
  * ``sigmaG_lims`` - The percentiles for sigmaG filtering (default of [25, 75]).
 
 
-Stamp Filtering
----------------
-
-The stamp filtering stage is only applied if the ``do_stamp_filter`` parameter is set to True. This stage creates a single stamp representing the sum, mean, median, or variance weighted mean of pixel values for the stamps at each time step. The stamp type is defined by the ``stamp_type`` parameter and can take on values ``median``, ``mean``, ``sum``, or ``weighted`` (for variance weighted). All of the stamp types drop masked pixels from their computations. The mean and median sums are computed over only the valid time steps from the light curve filtering phase (dropping stamps with outlier fluxes). The sum coadd uses all the time steps regardless of the first phase of filtering.
-
-The stamps are filtered based on how closely the pixel values in the stamp image represent a Gaussian defined with the parameters:
-
-* ``center_thresh`` - The percentage of flux in the central pixel. For example setting this to 0.9 will require that the central pixel of the stamp has 90 percent of all the flux in the stamp. 
-* ``peak_offset`` - How far the brightest pixel is from the center of the stamp (in pixels). For example a peak offset of [2.0, 3.0] requires that the brightest pixel in the stamp is at most 2 pixels from the center in the x-dimension and 3-pixels from the center in the y-dimension.
-* ``mom_lims`` - Compute the Gaussian moments of the image and compares them to the thresholds.
-
-Relevant stamp filtering parameters include:
-
- * ``center_thresh`` - The percentage of flux in the central pixel.
- * ``chunk_size`` - The number of candidate trajectories to filter in a batch. Used to control memory usage.
- * ``do_stamp_filter`` - A Boolean indicating whether to generate and filter stamps.
- * ``peak_offset`` - A length 2 list indicating how far the peak is from the center of the stamp in each of the x and y dimensions.
- * ``mom_lims`` -  A length 5 list providing thresholds on the images moments.
- * ``stamp_radius`` - The radius of the stamps to use.
-
-Note that stamps are only generated and output into files if ``do_stamp_filter`` is set to true.
-
-
 Clustering
 ----------
 

docs/source/user_manual/search_params.rst

@@ -33,11 +33,6 @@ Configuration Parameters
 +------------------------+-----------------------------+----------------------------------------+
 | **Parameter**          | **Default Value**           | **Interpretation**                     |
 +------------------------+-----------------------------+----------------------------------------+
-| ``center_thresh``      | 0.00                        | The minimum fraction of total flux     |
-|                        |                             | within a stamp that must be contained  |
-|                        |                             | in the central pixel                   |
-|                        |                             | (if ``do_stamp_filter=True``).         |
-+------------------------+-----------------------------+----------------------------------------+
 | ``chunk_size``         | 500000                      | The batch size to use when processing  |
 |                        |                             | the results of the on-GPU search.      |
 +------------------------+-----------------------------+----------------------------------------+
@@ -76,9 +71,6 @@ Configuration Parameters
 +------------------------+-----------------------------+----------------------------------------+
 | ``do_mask``            | True                        | Apply the mask to the raw pixels.      |
 +------------------------+-----------------------------+----------------------------------------+
-| ``do_stamp_filter``    | True                        | Apply post-search filtering on the     |
-|                        |                             | image stamps.                          |
-+------------------------+-----------------------------+----------------------------------------+
 | ``encode_num_bytes``   | -1                          | The number of bytes to use to encode   |
 |                        |                             | ``psi`` and ``phi`` images on GPU. By  |
 |                        |                             | default a ``float`` encoding is used.  |
@@ -113,20 +105,10 @@ Configuration Parameters
 |                        |                             | computed likelihood above this         |
 |                        |                             | threshold are rejected.                |
 +------------------------+-----------------------------+----------------------------------------+
-| ``mom_lims``           | [35.5, 35.5, 2.0, 0.3, 0.3] | Thresholds for the moments of a        |
-|                        |                             | Gaussian fit to the flux, specified as |
-|                        |                             | ``[xx, yy, xy, x, y]``.                |
-|                        |                             | If ``do_stamp_filter=True``.           |
-+------------------------+-----------------------------+----------------------------------------+
 | ``num_obs``            | 10                          | The minimum number of non-masked       |
 |                        |                             | observations for the object to be      |
 |                        |                             | accepted.                              |
 +------------------------+-----------------------------+----------------------------------------+
-| ``peak_offset``        | [2.0, 2.0]                  | How far, in pixels, the brightest pixel|
-|                        |                             | in the stamp can be from the central   |
-|                        |                             | pixel in each direction ``[x,y]``.     |
-|                        |                             | If ``do_stamp_filter=True``).          |
-+------------------------+-----------------------------+----------------------------------------+
 | ``psf_val``            | 1.4                         | The value for the standard deviation of|
 |                        |                             | the point spread function (PSF).       |
 +------------------------+-----------------------------+----------------------------------------+
@@ -148,9 +130,8 @@ Configuration Parameters
 |                        |                             | creating a stamp for stamp filtering   |
 |                        |                             | (in pixels).                           |
 +------------------------+-----------------------------+----------------------------------------+
-| ``stamp_type``         | sum                         | The type of coadd to use during stamp  |
-|                        |                             | filtering (if ``do_stamp_filter=True``)|
-|                        |                             | if:                                    |
+| ``stamp_type``         | sum                         | The type of coadd to use as the main   |
+|                        |                             | stamp:                                 |
 |                        |                             | * ``sum`` - (default) Per pixel sum    |
 |                        |                             | * ``median`` - Per pixel median        |
 |                        |                             | * ``mean`` - Per pixel mean            |

src/kbmod/configuration.py

@@ -17,7 +17,6 @@ def __init__(self):
         self._required_params = set()
 
         self._params = {
-            "center_thresh": 0.00,
             "chunk_size": 500000,
             "clip_negative": False,
             "cluster_eps": 20.0,
@@ -27,7 +26,6 @@ def __init__(self):
             "debug": False,
             "do_clustering": True,
             "do_mask": True,
-            "do_stamp_filter": True,
             "encode_num_bytes": -1,
             "generator_config": {
                 "name": "EclipticCenteredSearch",
@@ -41,9 +39,7 @@ def __init__(self):
             "im_filepath": None,
             "lh_level": 10.0,
             "max_lh": 1000.0,
-            "mom_lims": [35.5, 35.5, 2.0, 0.3, 0.3],
             "num_obs": 10,
-            "peak_offset": [2.0, 2.0],
             "psf_val": 1.4,
             "result_filename": None,
             "results_per_pixel": 8,

src/kbmod/filters/stamp_filters.py

@@ -63,29 +63,10 @@ def extract_search_parameters_from_config(config):
     else:
         raise ValueError(f"Unrecognized stamp type: {stamp_type}")
 
-    # Filtering parameters (with validity checking)
-    params.do_filtering = config["do_stamp_filter"]
-    params.center_thresh = config["center_thresh"]
-
-    peak_offset = config["peak_offset"]
-    if len(peak_offset) != 2:
-        raise ValueError(f"Expected length 2 list for peak_offset. Found {peak_offset}")
-    params.peak_offset_x = peak_offset[0]
-    params.peak_offset_y = peak_offset[1]
-
-    mom_lims = config["mom_lims"]
-    if len(mom_lims) != 5:
-        raise ValueError(f"Expected length 5 list for mom_lims. Found {mom_lims}")
-    params.m20_limit = mom_lims[0]
-    params.m02_limit = mom_lims[1]
-    params.m11_limit = mom_lims[2]
-    params.m10_limit = mom_lims[3]
-    params.m01_limit = mom_lims[4]
-
     return params
 
 
-def get_coadds_and_filter_results(result_data, im_stack, stamp_params, chunk_size=1_000_000, colname="stamp"):
+def make_coadds(result_data, im_stack, stamp_params, chunk_size=1_000_000, colname="stamp"):
     """Create the co-added postage stamps and filter them based on their statistical
      properties. Results with stamps that are similar to a Gaussian are kept.
 
@@ -157,17 +138,12 @@ def get_coadds_and_filter_results(result_data, im_stack, stamp_params, chunk_siz
         # collecting RawImage but leaving a dangling ref to the attribute.
         # That's a fix for another time so I'm leaving it as a copy here
         for ind, stamp in enumerate(stamps_slice):
-            if stamp.width > 1:
-                stamps_to_keep.append(np.array(stamp.image))
-                keep_row[start_idx + ind] = True
+            stamps_to_keep.append(np.array(stamp.image))
+            keep_row[start_idx + ind] = True
 
         # Move to the next chunk.
         start_idx += chunk_size
 
-    # Do the actual filtering of results
-    if stamp_params.do_filtering:
-        result_data.filter_rows(keep_row, label="stamp_filter")
-
     # Append the coadded stamps to the results. We do this after the filtering
     # so we are not adding a jagged array.
     result_data.table[colname] = np.array(stamps_to_keep)
@@ -195,7 +171,6 @@ def append_coadds(result_data, im_stack, coadd_types, radius, chunk_size=100_000
 
     params = StampParameters()
     params.radius = radius
-    params.do_filtering = False
 
     # Loop through all the coadd types in the list, generating a corresponding stamp.
     for coadd_type in coadd_types:
@@ -213,7 +188,7 @@ def append_coadds(result_data, im_stack, coadd_types, radius, chunk_size=100_000
             raise ValueError(f"Unrecognized stamp type: {coadd_type}")
 
         # Do the generation (without filtering).
-        get_coadds_and_filter_results(
+        make_coadds(
             result_data,
             im_stack,
             params,

src/kbmod/run_search.py

@@ -9,7 +9,12 @@
 from .configuration import SearchConfiguration
 from .filters.clustering_filters import apply_clustering
 from .filters.sigma_g_filter import apply_clipped_sigma_g, SigmaGClipping
-from .filters.stamp_filters import append_all_stamps, append_coadds, get_coadds_and_filter_results
+from .filters.stamp_filters import (
+    append_all_stamps,
+    append_coadds,
+    extract_search_parameters_from_config,
+    make_coadds,
+)
 
 from .results import Results
 from .trajectory_generator import create_trajectory_generator
@@ -238,11 +243,6 @@ def run_search(
             trj_generator = create_trajectory_generator(config, work_unit=None)
         keep = self.do_gpu_search(config, stack, trj_generator)
 
-        if config["do_stamp_filter"]:
-            stack.copy_to_gpu()
-            get_coadds_and_filter_results(keep, stack, config)
-            stack.clear_from_gpu()
-
         if config["do_clustering"]:
             cluster_timer = kb.DebugTimer("clustering", logger)
             mjds = [stack.get_obstime(t) for t in range(stack.img_count())]
@@ -255,12 +255,14 @@ def run_search(
             apply_clustering(keep, cluster_params)
             cluster_timer.stop()
 
-        # Generate additional coadded stamps without filtering.
+        # Generate coadded stamps without filtering -- both the "stamp" column
+        # as well as any additional coadds.
+        stamp_params = extract_search_parameters_from_config(config)
+        make_coadds(keep, stack, stamp_params, colname="stamp")
         if len(config["coadds"]) > 0:
             stack.copy_to_gpu()
             append_coadds(keep, stack, config["coadds"], config["stamp_radius"])
             stack.clear_from_gpu()
-        logger.info(f"Found {len(keep)} potential trajectories.")
 
         # Extract all the stamps for all time steps and append them onto the result rows.
         if config["save_all_stamps"]:

src/kbmod/search/bindings.cpp

@@ -42,7 +42,6 @@ PYBIND11_MODULE(search, m) {
     search::stack_search_bindings(m);
     search::stamp_creator_bindings(m);
     search::trajectory_bindings(m);
-    search::image_moments_bindings(m);
     search::stamp_parameters_bindings(m);
     search::psi_phi_array_binding(m);
     search::debug_timer_binding(m);

src/kbmod/search/common.h

@@ -142,50 +142,9 @@ struct SearchParameters {
 struct StampParameters {
     int radius = 10;
     StampType stamp_type = STAMP_SUM;
-    bool do_filtering = false;
-
-    // Thresholds on the location of the image peak.
-    float center_thresh;
-    float peak_offset_x;
-    float peak_offset_y;
-
-    // Limits on the moments.
-    float m01_limit;
-    float m10_limit;
-    float m11_limit;
-    float m02_limit;
-    float m20_limit;
-
-    const std::string to_string() const {
-        // If filtering is turned off, output the minimal information on a single line.
-        // Otherwise dump the full statistics on multiple lines.
-        if (!do_filtering) {
-            return ("Type: " + std::to_string(stamp_type) + "  Radius: " + std::to_string(radius) +
-                    "  Filtering: false");
-        } else {
-            return ("Type: " + std::to_string(stamp_type) + "\nRadius: " + std::to_string(radius) +
-                    "\nFiltering: true" + "\nCenter Thresh: " + std::to_string(center_thresh) +
-                    "\nPeak Offset: x=" + std::to_string(peak_offset_x) + " y=" +
-                    std::to_string(peak_offset_y) + "\nMoment Limits: m01=" + std::to_string(m01_limit) +
-                    " m10=" + std::to_string(m10_limit) + " m11=" + std::to_string(m11_limit) +
-                    " m02=" + std::to_string(m02_limit) + " m20=" + std::to_string(m02_limit));
-        }
-    }
-};
-
-// Basic image moments use for analysis.
-struct ImageMoments {
-    float m00;
-    float m01;
-    float m10;
-    float m11;
-    float m02;
-    float m20;
 
     const std::string to_string() const {
-        return "m00: " + std::to_string(m00) + " m01: " + std::to_string(m01) +
-               " m10: " + std::to_string(m10) + " m11: " + std::to_string(m11) +
-               " m02: " + std::to_string(m02) + " m20: " + std::to_string(m20);
+        return ("Type: " + std::to_string(stamp_type) + "  Radius: " + std::to_string(radius));
     }
 };
 
@@ -225,33 +184,12 @@ static void trajectory_bindings(py::module &m) {
                     }));
 }
 
-static void image_moments_bindings(py::module &m) {
-    py::class_<ImageMoments>(m, "ImageMoments", pydocs::DOC_ImageMoments)
-            .def(py::init<>())
-            .def("__str__", &ImageMoments::to_string)
-            .def_readwrite("m00", &ImageMoments::m00)
-            .def_readwrite("m01", &ImageMoments::m01)
-            .def_readwrite("m10", &ImageMoments::m10)
-            .def_readwrite("m11", &ImageMoments::m11)
-            .def_readwrite("m02", &ImageMoments::m02)
-            .def_readwrite("m20", &ImageMoments::m20);
-}
-
 static void stamp_parameters_bindings(py::module &m) {
     py::class_<StampParameters>(m, "StampParameters", pydocs::DOC_StampParameters)
             .def(py::init<>())
             .def("__str__", &StampParameters::to_string)
             .def_readwrite("radius", &StampParameters::radius)
-            .def_readwrite("stamp_type", &StampParameters::stamp_type)
-            .def_readwrite("do_filtering", &StampParameters::do_filtering)
-            .def_readwrite("center_thresh", &StampParameters::center_thresh)
-            .def_readwrite("peak_offset_x", &StampParameters::peak_offset_x)
-            .def_readwrite("peak_offset_y", &StampParameters::peak_offset_y)
-            .def_readwrite("m01_limit", &StampParameters::m01_limit)
-            .def_readwrite("m10_limit", &StampParameters::m10_limit)
-            .def_readwrite("m11_limit", &StampParameters::m11_limit)
-            .def_readwrite("m02_limit", &StampParameters::m02_limit)
-            .def_readwrite("m20_limit", &StampParameters::m20_limit);
+            .def_readwrite("stamp_type", &StampParameters::stamp_type);
 }
 
 #endif /* Py_PYTHON_H */

src/kbmod/search/pydocs/raw_image_docs.h

@@ -205,50 +205,6 @@ static const auto DOC_RawImage_compute_bounds = R"doc(
       A ``(min, max)`` tuple.
   )doc";
 
-static const auto DOC_RawImage_find_peak = R"doc(
-  Returns the pixel coordinates of the maximum value.
-
-  Parameters
-  ----------
-  furthest_from_center : `bool`
-      When `True`, and multiple identical maxima exist, returns the one that is
-      at a greatest L2 distance from the center of the image. Otherwise it
-      returns the last maxima that was found in a row-wise ordered search.
-
-  Returns
-  -------
-  location : `Index`, optional
-      Index of the maximum.
-  )doc";
-
-static const auto DOC_RawImage_find_central_moments = R"doc(
-  Returns the central moments of the image.
-
-  Returns
-  -------
-  moments : `ImageMoments`
-      Image moments.
-  )doc";
-
-static const auto DOC_RawImage_center_is_local_max = R"doc(
-  A filter on whether the center of the stamp is a local
-    maxima and the percentage of the stamp's total flux in this
-    pixel.
-
-  Parameters
-  ----------
-  local_max : ``bool``
-    Require the central pixel to be a local maximum.
-  flux_thresh : ``float``
-    The fraction of the stamp's total flux that needs to be in
-    the center pixel [0.0, 1.0].
-
-  Returns
-  -------
-  keep_row : `bool`
-      Whether or not the stamp passes the check.
-  )doc";
-
 static const auto DOC_RawImage_create_stamp = R"doc(
   Create an image stamp around a given region.
 

src/kbmod/search/pydocs/stamp_creator_docs.h

@@ -141,25 +141,6 @@ static const auto DOC_StampCreator_get_coadded_stamps = R"doc(
 
   )doc";
 
-static const auto DOC_StampCreator_filter_stamp = R"doc(
-  Filters stamps based on the given parameters.
-      
-  Applies the following filters: peak position, percent flux at central pixel,
-  and image moments.
-
-  Parameters
-  ----------
-  img : `RawImage`
-      The image to test.
-  params : `StampParameters`
-      The parameters for stamp generation and filtering.
-
-  Returns
-  -------
-  `bool`
-      Whether or not to filter the stamp.
-  )doc";
-
 static const auto DOC_StampCreator_create_variance_stamps = R"doc(
   Create a vector of stamps from the variance layer centered on the
   predicted position of an Trajectory at different times.