Minimal CLI

[cisaacstern]

A first draft of CLI functionality related to pangeo-forge/roadmap#37. Local install can be made by running poetry install from the repo root, following the recommendation of https://typer.tiangolo.com/tutorial/package/.

The top-level entry point is

pangeo-forge --help

➜ pangeo-forge --help
Usage: pangeo-forge [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion  Install completion for the current shell.
  --show-completion     Show completion for the current shell, to copy it or
                        customize the installation.
  --help                Show this message and exit.

Commands:
  bakery
  catalog
  lint     Lint a recipe

where bakery and catalog are typer sub-applications and lint is a top-level function. (Design inspired by pangeo-forge/pangeo-forge-recipes#69; specifics subject to change.)

Currently the bakery sub-application is the only functional component

pangeo-forge bakery --help

➜ pangeo-forge bakery --help
Usage: pangeo-forge bakery [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  ls  List available bakeries and associated build-logs.

Calling ls from this sub-application without arguments returns the list of bakeries from pangeo-forge/bakery-database

 ➜ pangeo-forge bakery ls       
['devseed.bakery.development.aws.us-west-2', 'devseed.bakery.development.azure.westeurope', 'great_bakery']

I've patched in great_bakery at runtime as a stand-in for the forthcoming pangeo-forge/pangeo-forge-gcs-bakery#19

Calling ls with one of the above-listed --bakery-ids returns details about that bakery

pangeo-forge bakery ls --bakery-id 'great_bakery'

➜ pangeo-forge bakery ls --bakery-id 'great_bakery'
{
    'targets': {
        'osn': {
            'anon': True,
            'client_kwargs': {'endpoint_url': 'https://ncsa.osn.xsede.org'},
            'root_path': 's3://Pangeo/pangeo-forge'
        }
    }
}

Further appending --view build-logs to this command fetches the build-logs.json file from the bakery's storage target and renders it as a table

As conceived here, build-logs.json is a JSON object which lives at at the root path of a bakery's storage bucket and records metadata about each dataset built to the bucket in which it resides. A further ADR and community discussion will be needed regarding whether to include such an object in Pangeo Forge and, if so, its specification and implementation details. Briefly, the purpose of the object would be to tie the provenance of datasets in a given storage location back to their corresponding feedstocks and recipes. Among other things, this aids cataloging, insofar as the cataloging tooling (which ideally can generate catalog items based on crawling storage buckets) will need to be aware of information in the feedstocks (i.e. meta.yamls).

pangeo-forge bakery ls --bakery-id 'great_bakery' --view build-logs

┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ Run ID ┃ Timestamp           ┃ Feedstock                ┃ Recipe           ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ 00006  │ 2021-10-01 00:00:00 │ soda342-feedstock@0.1    │ recipe           │
│ 00005  │ 2021-09-30 00:00:00 │ orca36-feedstock@0.1     │ recipes:surf-fma │
│ 00004  │ 2021-09-29 00:00:00 │ hycom50-feedstock@0.1    │ recipes:int-aso  │
│ 00003  │ 2021-09-28 00:00:00 │ gigatl-feedstock@0.1     │ recipes:surf-aso │
│ 00002  │ 2021-09-27 00:00:00 │ fesom-feedstock@0.1      │ recipes:surf-fma │
│ 00001  │ 2021-09-26 00:00:00 │ enatl60-feedstock@0.1    │ recipes:int-aso  │
│ 00000  │ 2021-09-25 00:00:00 │ noaa-oisst-feedstock@0.1 │ recipe           │
└────────┴─────────────────────┴──────────────────────────┴──────────────────┘

s3://Pangeo/pangeo-forge/build-logs.json is a real object I put on OSN for illustration/testing purposes, which is actually loaded at runtime, but feedstock and recipe names here are imaginary.

Finally, the logs can be filtered based on a specific feedstock name (or substring thereof)

pangeo-forge bakery ls --bakery-id 'great_bakery' --view build-logs --feedstock-name gigatl

┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ Run ID ┃ Timestamp           ┃ Feedstock                ┃ Recipe           ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ 00003  │ 2021-09-28 00:00:00 │ gigatl-feedstock@0.1     │ recipes:surf-aso │
└────────┴─────────────────────┴──────────────────────────┴──────────────────┘

[cisaacstern]

(Converted this to draft; will re-mark as Ready for review once I've incorporated all comments from first review cycle.)

[cisaacstern]

Thanks to all for your feedback on this. Closing because the design direction has been superseded by #6. The effort here (both in code and reviews) is not lost: I'll certainly be referring back to it extensively in the coming weeks.