variadic wsclean outputs

[o-smirnov]

So I've got this in my recipe, and it works, but it's a hack and it bothers me:

    image5:
      _use: lib.steps.wsclean.image_pol
      params:
        ms: "{steps.image.ms}"
        prefix: '{recipe.dir_out}/im5/im5{recipe.suffix}-int-s{recipe.scan}-{recipe.dd_sol_type}'
        column: "{previous.out_column}"
        fits_mask: "{steps.mask3.mask}"
        intervals_out: 4
        niter: 100000
        no_update_model_required: true

    mean_image:
      cab: fitstool
      params:
        mean: true
        images: "{previous.prefix}-t????-MFS-I-image.fits"
        output: "{previous.prefix}-mean-MFS-I-image.fits" 

I'm using wsclean to image per-interval images, then using fitstool to make a mean image. Ideally, the second step should have an images input of "a list of the per-interval Stokes I MFS images". The glob and the substitution above achieves that effect, but this is not robust, as now the mean_image step is relying on a particular filename convention to come out of the image5 step.

Ideally, the wsclean cab should define a formal output of "the set of per-interval Stokes I MFS images". The problem is, the filename conventions (and even the semantics) change depending on the mode you run wsclean in. If you've got channels-out > 1, it will produce *-00??-* and MFS images. If you've got polarization, these will also have IQUV suffixes. If you've got intervals-out set, they will have *-t00??-* prefixes. Any combination of the above is possible, so that gives us 2^3 output naming conventions. Simple globs and substitutions are not rich enough to capture this logic.

So far, I see two ways to solve this:

• have 8 different wsclean_xxx cab definitions for the 8 different combinations of output modes. This seems repetitive and error-prone (although with the use of _use, we can probably derive them all from a base definition and only change the relevant bits, so maybe it's not so bad).

• add support for an optional .py module to go with the cab definition. This would have a predefined function to modify the content of the outputs dict based on the settings of the inputs dict. This seems too complicated, but is probably the most general solution...

Not that thrilled with either so maybe @SpheMakh has a better idea?

[o-smirnov]

@SpheMakh if you're in a Stimela frame of mind, start by thinking about this.

[o-smirnov]

On the bright side, I just inserted a step like this into my recipe, and it simply works...

    mean_cubes:
      recipe:
        name: mean_iquv
        inputs:
          prefix:
            dtype: str
        for_loop:
          var: stokes
          over: [I, Q, U, V]
        steps:
          mean:
            cab: fitstool
            params:
              mean: true
              images: "{recipe.prefix}-t????-MFS-{recipe.stokes}-image.fits"
              output: "{recipe.prefix}-mean-MFS-{recipe.stokes}-image.fits" 
      params:
        prefix: "{previous.prefix}"

[o-smirnov]

A clean-but-wordy solution to this is to support a required: false option for outputs, and just list all possible combinations in the schema:

  model_cube_stokes:
    info: Model images
    dtype: List[File]
    implicit: "{current.prefix}-[0-9][0-9][0-9][0-9]-[IQUV]-model.fits"
    required: false
  model_mfs_stokes:
    info: Model images
    dtype: List[File]
    implicit: "{current.prefix}-MFS-[IQUV]-model.fits"
    required: false
  model_mfs:
    info: Model image
    dtype: File
    implicit: "{current.prefix}-MFS-model.fits"
    required: false
  model:
    info: Model images
    dtype: File
    implicit: "{current.prefix}-model.fits"
    required: false
  model_interval_cube_stokes:
    info: Model images
    dtype: List[File]
    implicit: "{current.prefix}-t[0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]-[IQUV]-model.fits"
    required: false
  model_interval_mfs_stokes:
    info: Model images
    dtype: List[File]
    implicit: "{current.prefix}-t[0-9][0-9][0-9][0-9]-MFS-[IQUV]-model.fits"
    required: false
  model_interval_mfs:
    info: Model image
    dtype: File
    implicit: "{current.prefix}-t[0-9][0-9][0-9][0-9]-MFS-model.fits"
    required: false
  model_interval:
    info: Model images
    dtype: File
    implicit: "{current.prefix}-t[0-9][0-9][0-9][0-9]--model.fits"
    required: false

Alternatively, we could add logic to the schema and put substitutions to work. For example, add a when_present section which, if set, would assign arbitrary variables in the current substitution namespace:

inputs:
    intervals_out:
        when_present:
            intervals_filename_prefix: "t[0-9][0-9][0-9][0-9]-"
            # ...

outputs:
    model_cube_stokes:
        info: Model images
        dtype: List[File]
        implicit: "{current.prefix}{current.intervals_filename_prefix}-[0-9][0-9][0-9][0-9]-[IQUV]-model.fits"