Cross referencing step parameters

[SpheMakh]

I already have this implemented in the usecwl branch RecipeStep.py file. This allows something like this

https://github.com/ratt-ru/Stimela/blob/8563da2e831a36bcd3c6164324e1632a18751616/examples/simulation_pipeline.py#L5-L28

[o-smirnov]

You could also do it with the same OmegaConf mechanism that allows config files to cross-reference each other:

• Define a schema for a recipe

• Dynamically load recipe configs into the global config object under recipe.RECIPE_NAME.blah.blah, as they are being defined

• Refer to other parameters as "{recipe.RECIPE_NAME.STEP.PARAMETER}"

• As a bonus, you can refer to stuff from the base and cabs files as "{base.IMAGE.blah.blah}"

I kind of like it because then we have a unified, hierarchical namespace for everything: configuration, image and cab definitions, recipe definitions, etc.

Also, from this, it's a trivial step to support recipes that are entirely defined as a YaML file.

[SpheMakh]

I kind of like it because then we have a unified, hierarchical namespace for everything: configuration, image and cab definitions, recipe definitions, etc.

I had something like this before but abandoned it because it was not easy to keep track of everything properly. OmegaConf does make this simpler

This is what's left of the old implementation. The idea was to allow a pipeline to be (re-)executed from JSon file

https://github.com/ratt-ru/Stimela/blob/eba6a69b953161d53e973d65f1d6fa74b923aeb4/stimela/recipe.py#L618-L635

[SpheMakh]

You could also do it with the same OmegaConf mechanism that allows config files to cross-reference each other:

Can you point me to where you do this?

[o-smirnov]

I don't think I do in the current codebase, but substituting {x.y.z} as a value is a standard OmegaConf feature, just try it in the current YaML files.

[o-smirnov]

Your stimela runcab (which indeed ought to be stimela exec I think!) gives me an idea. OmegaConf can be told to load a config section up from command-line arguments. You can combine this with exec very simply to run a cab (i.e. essentially a single-step recipe) from the command line.

[SpheMakh]

But here's another feature you can't have with argparse or click (at least easily) -- overriding recipe variables from the command line. stimela exec recipe my_selfcal_recipe.yml inputs.ms=foo.ms vars.size=1024 vars.scale=1arcsec.

Agreed. Let's not break our backs over this. Let's go for foo=VALUE

[o-smirnov]

An excellent choice! You will not regret it.

Also, in this case we can simplify the invocation further by auto-detecting if the first argument to exec is a recipe file or a cab name. stimela exec [recipe.yml|cabname] foo=VALUE ....

[o-smirnov]

Yep, click is neat as fsck. I pushed some changes -- take a look at build or exxec, for example. Everything is so much more succint.

[o-smirnov]

While we're at it, should this command not be unified with stimela run recipe.py?

[SpheMakh]

I thought it was neater to separate recipes that are scripts and those that are yaml files, but there is no fundamental difference between the two; both are just recipes.