CLI for preparing / validating recipes

[rabernat]

User Profile

As recipe contributor

User Action

I want to test my recipe on the command line before submitting a pull request

User Goal

so that I can avoid a slow debugging cycle talking to the pangeo forge bot on github. (Example: pangeo-forge/staged-recipes#150)

Acceptance Criteria

I run a command like

$ pangeo-forge recipe validate recipe_folder/

and see output like

It looks like your meta.yaml does not conform to the specification.

            1 validation error for MetaYaml
    pangeo_notebook_version
      field required (type=value_error.missing)

or

When I tried to import your recipe module, I encountered this error

            line 17, in <module>
        fs.ls(url_base + str(year), detail=False)
    NameError: name 'fs' is not defined

Please correct your recipe module so that it's importable.

Linked Issues

No response

[derekocallaghan]

Hi @rabernat, as discussed in the Oct 24 Pangeo Forge meeting, here are a few thoughts about what might be required for implementation. These are based on recent local debugging of staged recipes. Although the Pangeo Forge documentation recommends locally running a pruned version of a recipe, this won't always detect common staged recipe issues e.g.

• As mentioned above:
  • meta.yaml issues
  • Recipe import issues
• Issues encountered when running a recipe with Beam:
  • Hash serialization
  • Pickler serialization
  • etc

As pangeo-forge-runner provides a CLI for feedstock loading and recipe baking, involving recipe loading, running with Beam etc., it might be useful to extend it to add a validate command such as the one you mentioned above. From looking through the code, the following changes/steps might be a starting point:

• Include pangeo-forge-runner with pangeo-forge-recipes:
  • This would lead to additional dependencies for pangeo-forge-recipes e.g. ruamel.yaml, Beam etc
  • This may not be desirable for everyone, but from my own experience in the last few weeks, pangeo-forge-runner feels like it belongs as a feature of pangeo-forge-recipes, unlike e.g. pangeo-forge-orchestrator.
• A new Validate command implementation would be added to the pangeo-forge-runner CLI, similar to pangeo_forge_runner.commands.bake.Bake
• Validate would create a config using the supplied parameters, for local bakery execution of a local directory containing the recipe module and meta.yaml. E.g. here's what I used for recent local recipe debugging:

{'Bake': {'bakery_class': 'pangeo_forge_runner.bakery.local.LocalDirectBakery',
  'recipe_id': 'eooffshore_ics_ccmp_v02_1_nrt_wind',
  'feedstock_subdir': 'recipes/eooffshore_ics_ccmp_v02_1_nrt_wind',
  'repo': 'https://github.com/eooffshore/staged-recipes',
  'ref': '663f30c95c406b9efe012b9bae66fa1f386b539b',
  'job_name': 'CCMP',
  'prune': True},
 'LocalDirectBakery': {'num_workers': 1},
 'TargetStorage': {'fsspec_class': 'fsspec.implementations.local.LocalFileSystem',
  'root_path': './ccmp.zarr'},
 'InputCacheStorage': {'fsspec_class': 'fsspec.implementations.local.LocalFileSystem',
  'root_path': './input-cache/'},
 'MetadataCacheStorage': {'fsspec_class': 'fsspec.implementations.local.LocalFileSystem',
  'root_path': './metadata-cache/'}}

• A config similar to the above would then be used to run a pruned Bake, e.g. Bake(config=Config(bconfig)).start(). If this successfully ran to completion, it would ensure that the meta.yaml has been successfully loaded, the recipe(s) imported by Feedstock, executed with local Beam ("multi_processing" mode) etc.
• A step would be added to load the resulting output e.g. Zarr store (I was doing this manually and performing some simple checks e.g. printing the time dimension)
• In the example config above, a Git repo has been specified. The Validate command would instead use the specified local recipe path (containing feedstock_subdir). However, it looks like local repos/recipes aren't currently supported by Bake, so this would need to implemented - I assume it's related to the use of contentproviders.Local in pangeo_forge_runner.commands.base.BaseCommand. I'd originally tried this locally but it didn't work due to an error I can't recall. It seems useful to support local recipe paths enabling validation prior to repo commits.
• Test cases would be created for the Validate implementation.
• The Pangeo Forge documentation would be updated to document use of the pangeo_forge_runner validate... command.

I should be free for next week's Pangeo Forge meeting call if you want to discuss the above suggestions.

[yuvipanda]

+1 for moving the tutorial to use pangeo-forge-runner to test things out vs what is there now. The runner project is much newer than recipes, so a lot of the split here is just 'where things are' rather than 'how it ought to be'

[derekocallaghan]

+1 for moving the tutorial to use pangeo-forge-runner to test things out vs what is there now. The runner project is much newer than recipes, so a lot of the split here is just 'where things are' rather than 'how it ought to be'

If you like, I can take a look at adding a Validate command I suggested above into pangeo-forge-runner, I've largely been doing this locally already with calls to Bake()...start().