💡 I'm thinking about...

[gwenwindflower]

This is the place for anything that's on your mind about dbt, analytics engineering or this repository. Share thoughts on best practices, propose a blog or an article, or share an interesting resource that you've found.

[dbeatty10]

I'm thinking about tooling that can merging two different YAML files together.

TL;DR

If there is a method to combine two YAML files together, then it can be used to combine human edits and machine-generated content together. 🧑‍🦱 + 🤖 = ❤️

Use cases

One use case (briefly described here):

• an automated loading pipeline has new source tables appear over time
• dbt-codegen is used to configure each new source within a new YAML file
• the (hypothetical) YAML merge functionality combines the dbt-codegen-erated YAML files

Another use case:

• a golden YAML config file has portions that are hand-curated (like descriptions)
• a dbt-codegen-erated YAML config fills in undocumented tables/columns
• (merge functionality goes here)

Yet another use case:

• suppose a migration process has a tool that is automatically translating some legacy data warehouse scripting into a dbt project
• like all good migration efforts, development is continuing in the legacy system at the same time as the new stuff is being stood up (i.e., there is is a long-ish period of overlap between legacy system and dbt-based system)
• if there is YAML merge functionality available, the migration team can safely make edits to the same YAML config files that the automated migration tool is writing to.

Crazy, not real use case:

• your dbt project/package is like an non-fungible token (NFT)
• there is a game in which you can breed dbt projects together to spawn child projects (à la crypto breeding games)
• the (hypothetical) YAML merge functionality could be used under the covers to help with the breeding

Merge strategies

Suppose you have two YAML files describing dbt resources:

• blue.yml
• orange.yml

(Without loss of generalization, these could be dbt projects too.)

Then we can imagine them being combined in ways similar to relations in SQL:

• union (append orange.yml to the end of blue.yml)
• except (remove matching content of orange.yml from blue.yml)
• intersect (only keep content common to both)
• inner join (same as intersect?)
• full outer join (composed of all content that appears in either or both)
• left outer join (add any content from orange.yml that doesn't already exist within blue.yml)

Of course, the devil is in the details of what would constitute "matching content", the order to render it, etc.