Tagging/Metadata Umbrella

[elijahbenizzy]

A few things we're thinking about and we'd like the community's input:

1. In Hamilton tags always have keys/values. Should we just allow keys? E.G.

@tag(“is_feature”, “is_final_product”, author=“elijah”)
@tag(is_feature=True, is_final_product=True, author=“elijah”)

If we allow the first, how do we store/query tags?

2. Should we allow non-string types for keys (E.G. booleans?). Currently we allow lists, which just means multiple values. But booleans might be nice. Note this is similar to (1) in that these often express "existence" with no value.

3. Which use-cases make the most sense for tags? Which should we support? So far:

• Tracking purpose of a node
• Tracking ownership of a node/other metadata
• Indicating as a part of a data product/contract
• Some indication of versioning
• We also hack tags in to support, say, schemas, although that's internal and likely to be come more first-class.

4. What do list tags mean? Is this legit?

@tag(date=["2024", "01", "01"]

Comment here with:

(1) Your use-cases
(2) Thoughts on any of the above/more questions!

We can break out into further discussions if multiple of these are interesting.

[zilto]

The @tag() feature should restrict supported data types to prevent stuffing logic in there. Tags should be useful external to execution (e.g., help with readability, documentation, querying nodes, versioning, etc.).

The grey zone is how tags relate to the lifecycle hooks. The benefit of hooks is that they don't alter a function; you can import and use them outside Hamilton. A similar logic is relevant for visualizations:

# functions.py

@tag(env="dev")
def feature_a():
   ...
   
@tag(env="prod", owner="zilto")
def feature_b():
   ...

# runner.py
import functions

# schemas are a special type of hook
env_schemas = dict(
  dev=DEV_SCHEMA,
  prod=PROD_SCHEMA,
)

owner_hooks = dict(
   zilto=ZILTO_HOOKS,
)

dr = (
  driver.Builder()
  .with_modules(functions)
  .with_hooks(env_schemas, owner_hooks)  # <- execution logic based on tags
  .build()
)
# would need to specify on what key the hooks are related

# it's simpler to have "flat" stylesheet instead of "nested" ones because
# of graphviz limitations
env_stylesheet = dict(
  prod=PROD_STYLESHEET,
  dev=ENV_STYLESHEET
) 

dr.display_all_functions(stylesheet=env_stylesheet)  # <- viz logic based on tags