Metadata Specification Proposal

[ZacharyWills]

Adding spec, notes and some examples for semi-reproducible cataloging of a RUN. This is a relatively flexible item, since no automation has been produced yet. Eventually existing tooling will make this spec more difficult to change.

Open to comments and discussion. As per CIROH Meeting on July 18th 2023.

[ZacharyWills]

@jameshalgren Can we add as many people as we can to consider how to tackle this? It's also related to Andy Wood & the protocols as model run diffs would be good for the proposal stage before they were run or to reproduce.

[ZacharyWills]

A good bit of feedback I got that I want to record here: There should be a way to mark a specific hash and file as a "benchmark" or part of Operational workflows.

[arpita0911patel]

Zach, possible to update README.md so that the existing content is still there along with the new?

[ZacharyWills]

Quick update to this:

In pursuit of the goal of having each run output a metadata file, Austin has made us a nice repo for generating the Merkel Tree: https://github.com/aaraney/ht

Adding that after validation, then we can add comparison once the data stream is up and running.

I'll add the spec for the configs, the configs, and their locations later on once the stream is up and running.

Thanks,
Zach

[aaraney]

Thanks for writing this up, @ZacharyWills! I left a fair number of comments that hopefully with further the conversation in a positive direction! Look forward to hearing back from you!

[aaraney]

I likely just dont know the context, so this might be a moot point, but what is a RUN_CONFIG? Can we just remove it? If not, can we provide some supporting text to say what a RUN_CONFIG is?

[aaraney]

I would suggest using curl instead of wget. curl is generally available by default.

[aaraney]

small typo! resproduce -> reproduce

[aaraney]

Not sure if this file was intended to make it into the proposal? It seems like some of this content could be reshaped a bit and make its way into the README instead? Maybe we do want a living notes document though. Although, a github discussion thread might be more appropriate and accomplish the same goal. Thoughts?

[aaraney]

General comments:

• probably want to add some kind of spec version identifier. or, more desirably, when we are happy with the spec, lock it down and guarantee that we will never remove a field.

• I really like that you havent said anything about it being in JSON or specifying the format. I think we should harp on that being a design goal (if it is one). So kind of like schema.org, where you can choose to represent this information in a variety of formats. While that might cause pain points early on when, for example, format translation tooling isnt great, I think in the long run less format coupling is desirable.

• Consider adding another top level metadata key (or record, whatever you want to call it) and add hash as a key with its record containing algo or type. So, in json, something like:

  {
    "meta": {
      "hash": {
        "algo": "sha256"
      }
    }
  }

  This will provide a way to add new metadata fields in the future (if desired) and make it clear what hash algorithm is being used. I can see in the future needing to move a different hashing algorithm.

[aaraney]

As for general formatting, what are your thoughts on switching to a table format similar to schema.org? A schema.org Thing is a good enough example. In our context, that could look like:

Model

Property	Expected Type	Description
realization	Realization	NextGen framework configuration file that specifies the unit(s) of work for a model engine run.
configuration	Configuration	BMI model formulation configuration file

[aaraney]

I figure Specification.md will undergo a fair amount of change as we iron out the details for records themselves. So, I am not going to go overboard commenting about it.

[aaraney]

I think we can remove this. It kind of implies that the listed fields need to be ordered (although, I think that is a stretch).

[aaraney]

Thanks for getting this going, @ZacharyWills! Having now stared at this for 30 minutes and gone back and forth about what comments I want to make, it seems clear to me that we could solve this problem in a lot of different ways. But, as will all solutions, they all have pros and cons. My biggest take away so far is that this initial proposal should set up future proposals for success. I think the way we do that is by going with the simplest data model that does not limit future additions in places where we want expansion.

To get to that place, I think we first need some guard rails on what the spec is not. I figure we wont be able to determine all the things it is to start, without using it, so it is likely easier to say what it isn't. So kind of like goals, but anti-goals if you will haha. The purpose of saying what the spec isn't is to create consensus and avoid scope creep and have something to point to when suggestions are brought to the table. Please disagree and challenge me on this. It may just be an idea in my head and not a good one!

My general thoughts so far are (numbered, so it is easier to respond. Not ordered):

1. I think we probably want to create a key within Model that is either files, assets, configuration (or something) that is a list of realization and configuration record types. This will allow specifying the realization file and all other BMI init config / other supporting files and their details.
2. I like the distinction of configuration, forcing, and hydrofabric. However, im not sold on realization needing its own category. To me, its just another configuration file. I think we could instead just have a configuration record type that has Name: str, Store: "fs" | "os" (this could be confusing, "filesystem" or "bucket" is probably fine), Path: str, Hash: str, Variant: "unknown" | "realization" | "cfe" | .... This way we can treat all configuration files the same.
3. Im not sold on Hydrofabric living within Forcing. As you mention in notes.md:

Firstly there must be a heirarchical underestanding of how we represent data and configurations...

I don't see it the same. I think you can make the case that you can have forcing over a given hydrofabric domain, so Forcing should live within Hydrofabric. I am just not sold that the relationship between the two needs hierarchy. Why not just have Hydrofabric and Forcing as top level keys? My same comment about having files or assets as a subkey of Forcings and Hydrofabric applies here too.
4. Where do files like stream flow nudging or restart fall into this?
5. Pedantic: I am not a huge fan of RUN_CONFIG. We don't use upper case snake case anywhere else, so I think RunConfig or run_config are preferred. Even just Config or another single word key would be "nicer." Thoughts?
6. Pedantic: use the same case format across keys (i.e. model and realization, not Model and realization). My personal preference is lowercase snake case (i.e. run_config), but I definitely have a bias from my python background lol. Point being, we should pick something and stick with it.
7. I mention this in a later comment in notes.md, but we probaly want to introduce a top level Metadata or Meta as a top level key and add a hash key with its record containing Algo or Type. So, in json, something like:

{
  "Meta": {
    "Hash": {
      "Algo": "sha256"
    }
  }
}

[ZacharyWills]

So the hierarchy is based off of the assumption that things within the workflow are somewhat dependent (generating forcings for only your selected hydrofabric is more efficient than always generating them everywhere) but there's a lot of discussion about that going on now here: lynker-spatial/hfsubsetCLI#28

I totally agree on the syntax changes and I'll put some work in to fix the proposal to reflect those insights. For the most part I just wanted something to get the ball rolling on how AWI/CIROH-members/OWP//Ops will work with these run artifacts and the relationship between their configs and the desired outputs. That's ultimately the goal of the metadata is to facilitate that discussion at first, then later on when we're more practiced we can start to talk about how to transition things.

Thanks again for taking a look at this and any input is so valuable at this point to get us going from 0 -> something.

[aaraney]

small typo! underestanding -> understanding

[arpita0911patel]

Uploading the discussion from working group 2 slack channel regarding model_type_name in realization file.