Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-46697: Rework the run signature for make_direct_warp #992

Merged
merged 12 commits into from
Nov 12, 2024

Conversation

arunkannawadi
Copy link
Member

No description provided.

@arunkannawadi arunkannawadi changed the title DM-46697: Make new warping tasks compatible with CalibrateImageTask outputs. DM-46697: Rework the run signature for make_direct_warp Nov 5, 2024
@arunkannawadi arunkannawadi force-pushed the tickets/DM-46697 branch 6 times, most recently from 013256c to 5f06444 Compare November 6, 2024 17:07
@arunkannawadi arunkannawadi marked this pull request as ready for review November 6, 2024 17:11

@classmethod
def getImage(cls) -> float:
return 0.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not totally thrilled with the idea that we'll be burning even a small number of compute cycles by adding zeros to images, but if this helps a lot in reducing complexity it's probably still worth it. Maybe after this ticket we could add a "apply" and "unapply" images to BackgroundList itself, and implement them to do nothing when the list is empty such that you could use a trivial BackgroundList instead. If that sounds good, please make a ticket.

Either way, it might be good to give this class a leading-underscore name, since it doesn't seem to be used outside of method bodies and type annotations (which are not so important here in pipe_tasks at present).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option might be to drop this class, allow process to accept None for its background list arguments regardless of what the config options are, and move the check for consistency between input arguments and configuration up to run.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I certainly didn't like the way the code read when I had to check if the backgrounds were None before calling getImage, but I am on board with any of the solutions here. I'll give it some more thought and am happy to just add it on this ticket.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, this is what I'm inclining towards:

  1. Keep the TrivialBackgroundList class, but with a leading underscore name and drop the getImage method.
  2. Add apply and unapply methods (names TBD) to WarpDetectorInputs, since the operations happen on the components of that dataclass. These would do raise a RuntimeError if the background is None (current behavior), do nothing if the background is a _TrivialBackgroundList and do addition/subtraction only otherwise. It has the advantage of keeping track of whether we are dealing with background subtracted/added image or not.


return self._exposure


class MakeDirectWarpConnections(
PipelineTaskConnections,
dimensions=("tract", "patch", "skymap", "instrument", "visit"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(preexisting, below): I think we can drop the calexpType connection template, and save ourselves a deprecation cycle for the config field it creates later.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow the second part of this comment - what config field does this create?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A connection template creates a config field alongside all the collection-name ones, e.g. config.connections.calexpType.

@@ -265,7 +314,7 @@ def bgSubtracted(self) -> bool:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(preexisting, many lines above): extra newline after the opening triple-quote for MAX_NUMBER_OF_NOISE_REALIZATIONS. But I also wonder whether it's worthwhile to have a config limit on the maximum.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here: seems like bgSubtracted should also depend on doApplyNewBackground as well, which might make being able to assign to it tricky. Or if the assertion is that doApplyNewBackground=True requires doRevertOldBackground=False (because the skyCorr always includes a revert of the calexp background internally), we should add comment to that effect here and check it validate.

Copy link
Member Author

@arunkannawadi arunkannawadi Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having a non-configurable maximum limit is somewhat intentional, to prevent too many noise realizations from being generated accidentally. I'd rather just let the numberOfNoiseRealizations take any non-negative integer value than setting a configurable maximum limit.

visit_id = exposures[0].dataId["visit"]
for _, warp_detector_input in inputs.items():
visit_id = warp_detector_input.data_id["visit"]
break # Just need the visit id from any one of the inputs.

# The warpExposure routine is expensive, and we do not want to call
# it twice (i.e., a second time for PSF-matched warps). We do not
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The i.e. parenthetical here seems out of place; should the example now be noise realizations or mask fractions instead?

)
return False

elif self.config.useVisitSummaryPsf:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an elif on the same condition as the original if. Is it supposed to be an else? I'm not actually sure it should exist, because if useVisitSummaryPsf is False, we should just not try to get a PSF from the visit summary.

# called directly.
raise RuntimeError(
f"doApplyNewBackground is False, but {detector_inputs.data_id} has a background_apply."
)

# Calibrate the (masked) image.
# This should likely happen even if visit_summary is None.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it should happen even if visit_summary is None.

@arunkannawadi arunkannawadi force-pushed the tickets/DM-46697 branch 2 times, most recently from efb8cae to 7d17b31 Compare November 9, 2024 03:19
@arunkannawadi arunkannawadi merged commit 29321ee into main Nov 12, 2024
2 checks passed
@arunkannawadi arunkannawadi deleted the tickets/DM-46697 branch November 12, 2024 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants