Creating and enforcing testing and documentation

[gwenwindflower]

@b-per has drafted an excellent guide to using pre-commit hooks, a tool called Tangata (and would it be worth covering this as well, similar purpose https://hub.getdbt.com/tnightengale/dbt_meta_testing/latest/ ?? ) all centered around deciding what levels of coverage you need to set in your project and then automating the enforcement of those standards. Feels like a guide to me!

via archived developer blog repo

[b-per]

Hi @dbt-labs/devhub team! Is it OK for me to create an issue for this Blog post? Cheers.

[b-per]

Answers to the question on the template:

1. What is the main problem you are solving? What is your solution?
   Testing and documentation are the backbones of a healthy and reliable dbt project and we recommend to do a unique and not null test on all models at the minimum. The problem is that for large projects where the rules have not been enforced from the beginning it can be difficult to understand what needs to be done to bring the project to a better state and to avoid future issues.

2. Why should the reader care about this problem? Why is your solution the right one?
   Models untested and underdocumented can bring confusion and lead to wrong data being made available to end users. Having an approach and tools to go through the process of fixing the lack of tests and enforcing tests going forward will make dbt users more confident about the quality of their pipeline and will allow them to focus on more value added tasks

3. Can you list the steps of your solution for the reader here?

• find the current models that should be documented/tested
  • Decide what needs to be documented/tested (opinionated)
  • Document/test the models that are lacking such information. This can be done manually in YAML (potentially time consuming and error prone) or using the tool Tangata created by Chris Jenkins
• setup automated tests to prevent the rules to be broken, for example leveraging pre-commit hooks

7. Are there any resources that helped inspire or inform your idea?
   Quite a few threads on the debt Slack that ask how to setup automated checks on the quality of a dbt project + personal experience of wanting to increase the test coverage of a big project without knowing where to start

8. Are there other existing solutions that solve the problem, and if so, how is this solution better or different?
   Tools like dbt_meta_testing also allow users to check if a dbt project is following some predefined rules. This article is not so much about defining a new solution to a problem but more how to stitch together tools that have been created by the community

[b-per]

With the FigJam outline:

[gwenwindflower]

Hey @b-per, this is fantastic -- I've got some thoughts/questions/feedback on the outline and how we fit Tangata in, but that's expected and exactly what we'll sort out collaboratively in FigJam and the issue. Go ahead and use the Create issue from discussion option in the right sidebar here and we'll get cooking. Great stuff! 🙏🏻 Thank you!

[b-per]

I created the issue. I didn't use "Create issue from discussion" though because it didn't let me pick the template for the promotion from Discussion to Issue

[gwenwindflower]

Ah dang, good to know, thanks! I'll see if there's anything I can do to fix that for the future 🙏🏻