Add dbt doc macros for mart_ntd

[tiffanychu90]

Description

Create the macros used for NTD data models that will help populate column descriptions. Start with the most common ones about NTD agency info, and leave the service (upt, vrm, etc) columns off for now, until we see 3+ instances.

Resolves #3521

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

How has this been tested?

Tested all tables that present in mart_ntd to make sure models build correctly, and also ran poetry run dbt docs generate to make sure dbt docs page can be generated successfully.

jovyan@jupyter-tiffanychu90 ~/data-infra/warehouse (ntd-doc-macros) $ poetry run dbt run -s +"models/mart/ntd/dim_annual_service_agencies.sql"
23:31:02  Running with dbt=1.5.1
23:31:06  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
23:31:06  Found 422 models, 973 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
23:31:07  
23:31:19  Concurrency: 8 threads (target='dev')
23:31:19  
23:31:19  1 of 2 START sql view model tiffany_staging.stg_ntd_annual_data__2022__service_by_agency  [RUN]
23:31:21  1 of 2 OK created sql view model tiffany_staging.stg_ntd_annual_data__2022__service_by_agency  [CREATE VIEW (0 processed) in 1.24s]
23:31:21  2 of 2 START sql table model tiffany_mart_ntd.dim_annual_service_agencies ...... [RUN]
23:31:24  2 of 2 OK created sql table model tiffany_mart_ntd.dim_annual_service_agencies . [CREATE TABLE (2.2k rows, 7.9 MiB processed) in 3.71s]
23:31:24  
23:31:24  Finished running 1 view model, 1 table model in 0 hours 0 minutes and 17.72 seconds (17.72s).
23:31:25  
23:31:25  Completed successfully
23:31:25  
23:31:25  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2
jovyan@jupyter-tiffanychu90 ~/data-infra/warehouse (ntd-doc-macros) $ poetry run dbt run -s +dim_annual_funding_sources
23:32:07  Running with dbt=1.5.1
23:32:10  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
23:32:11  Found 422 models, 973 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
23:32:11  
23:32:14  Concurrency: 8 threads (target='dev')
23:32:14  
23:32:14  1 of 4 START sql view model tiffany_staging.stg_ntd_annual_data__2022__funding_sources_federal  [RUN]
23:32:14  2 of 4 START sql view model tiffany_staging.stg_ntd_annual_data__2022__funding_sources_local  [RUN]
23:32:14  3 of 4 START sql view model tiffany_staging.stg_ntd_annual_data__2022__funding_sources_state  [RUN]
23:32:15  1 of 4 OK created sql view model tiffany_staging.stg_ntd_annual_data__2022__funding_sources_federal  [CREATE VIEW (0 processed) in 0.74s]
23:32:15  2 of 4 OK created sql view model tiffany_staging.stg_ntd_annual_data__2022__funding_sources_local  [CREATE VIEW (0 processed) in 0.75s]
23:32:15  3 of 4 OK created sql view model tiffany_staging.stg_ntd_annual_data__2022__funding_sources_state  [CREATE VIEW (0 processed) in 0.79s]
23:32:15  4 of 4 START sql table model tiffany_mart_ntd.dim_annual_funding_sources ....... [RUN]
23:32:19  4 of 4 OK created sql table model tiffany_mart_ntd.dim_annual_funding_sources .. [CREATE TABLE (6.8k rows, 7.1 MiB processed) in 4.12s]
23:32:19  
23:32:19  Finished running 3 view models, 1 table model in 0 hours 0 minutes and 8.21 seconds (8.21s).
23:32:19  
23:32:19  Completed successfully
23:32:19  
23:32:19  Done. PASS=4 WARN=0 ERROR=0 SKIP=0 TOTAL=4
jovyan@jupyter-tiffanychu90 ~/data-infra/warehouse (ntd-doc-macros) $ poetry run dbt run -s +dim_annual_ntd_agency_information
23:33:01  Running with dbt=1.5.1
23:33:03  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
23:33:04  Found 422 models, 973 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
23:33:04  
23:33:07  Concurrency: 8 threads (target='dev')
23:33:07  
23:33:07  1 of 2 START sql view model tiffany_staging.stg_ntd__annual_database_agency_information  [RUN]
23:33:08  1 of 2 OK created sql view model tiffany_staging.stg_ntd__annual_database_agency_information  [CREATE VIEW (0 processed) in 1.25s]
23:33:08  2 of 2 START sql view model tiffany_mart_ntd.dim_annual_ntd_agency_information . [RUN]
23:33:10  2 of 2 OK created sql view model tiffany_mart_ntd.dim_annual_ntd_agency_information  [CREATE VIEW (0 processed) in 1.46s]
23:33:10  
23:33:10  Finished running 2 view models in 0 hours 0 minutes and 6.06 seconds (6.06s).
23:33:10  
23:33:10  Completed successfully
23:33:10  
23:33:10  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2
jovyan@jupyter-tiffanychu90 ~/data-infra/warehouse (ntd-doc-macros) $ poetry run dbt run -s +dim_annual_service_mode_time_periods
23:33:25  Running with dbt=1.5.1
23:33:28  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
23:33:28  Found 422 models, 973 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
23:33:28  
23:33:32  Concurrency: 8 threads (target='dev')
23:33:32  
23:33:32  1 of 2 START sql view model tiffany_staging.stg_ntd_annual_data__2022__service_by_mode_and_time_period  [RUN]
23:33:33  1 of 2 OK created sql view model tiffany_staging.stg_ntd_annual_data__2022__service_by_mode_and_time_period  [CREATE VIEW (0 processed) in 0.98s]
23:33:33  2 of 2 START sql table model tiffany_mart_ntd.dim_annual_service_mode_time_periods  [RUN]
23:33:37  2 of 2 OK created sql table model tiffany_mart_ntd.dim_annual_service_mode_time_periods  [CREATE TABLE (9.2k rows, 50.6 MiB processed) in 4.38s]
23:33:37  
23:33:37  Finished running 1 view model, 1 table model in 0 hours 0 minutes and 8.66 seconds (8.66s).
23:33:38  
23:33:38  Completed successfully
23:33:38  
23:33:38  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2
jovyan@jupyter-tiffanychu90 ~/data-infra/warehouse (ntd-doc-macros) $ poetry run dbt run -s +dim_monthly_ntd_ridership_with_adjustments
23:33:55  Running with dbt=1.5.1
23:33:57  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
23:33:58  Found 422 models, 973 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
23:33:58  
23:34:01  Concurrency: 8 threads (target='dev')
23:34:01  
23:34:01  1 of 11 START sql table model tiffany_staging.int_ntd__modes ................... [RUN]
23:34:01  2 of 11 START sql table model tiffany_staging.stg_ntd__monthly_ridership_with_adjustments_upt  [RUN]
23:34:01  3 of 11 START sql table model tiffany_staging.stg_ntd__monthly_ridership_with_adjustments_voms  [RUN]
23:34:01  4 of 11 START sql table model tiffany_staging.stg_ntd__monthly_ridership_with_adjustments_vrh  [RUN]
23:34:01  5 of 11 START sql table model tiffany_staging.stg_ntd__monthly_ridership_with_adjustments_vrm  [RUN]
23:34:03  1 of 11 OK created sql table model tiffany_staging.int_ntd__modes .............. [CREATE TABLE (22.0 rows, 374.0 Bytes processed) in 2.41s]
23:34:05  5 of 11 OK created sql table model tiffany_staging.stg_ntd__monthly_ridership_with_adjustments_vrm  [CREATE TABLE (2.3k rows, 72.2 MiB processed) in 4.47s]
23:34:05  6 of 11 START sql table model tiffany_staging.int_ntd__monthly_ridership_with_adjustments_vrm  [RUN]
23:34:06  3 of 11 OK created sql table model tiffany_staging.stg_ntd__monthly_ridership_with_adjustments_voms  [CREATE TABLE (2.3k rows, 64.6 MiB processed) in 4.52s]
23:34:06  7 of 11 START sql table model tiffany_staging.int_ntd__monthly_ridership_with_adjustments_voms  [RUN]
23:34:06  2 of 11 OK created sql table model tiffany_staging.stg_ntd__monthly_ridership_with_adjustments_upt  [CREATE TABLE (2.3k rows, 71.5 MiB processed) in 4.86s]
23:34:06  8 of 11 START sql table model tiffany_staging.int_ntd__monthly_ridership_with_adjustments_upt  [RUN]
23:34:06  4 of 11 OK created sql table model tiffany_staging.stg_ntd__monthly_ridership_with_adjustments_vrh  [CREATE TABLE (2.3k rows, 69.6 MiB processed) in 4.91s]
23:34:06  9 of 11 START sql table model tiffany_staging.int_ntd__monthly_ridership_with_adjustments_vrh  [RUN]
23:34:13  6 of 11 OK created sql table model tiffany_staging.int_ntd__monthly_ridership_with_adjustments_vrm  [CREATE TABLE (622.2k rows, 2.9 MiB processed) in 7.84s]
23:34:14  8 of 11 OK created sql table model tiffany_staging.int_ntd__monthly_ridership_with_adjustments_upt  [CREATE TABLE (622.2k rows, 2.9 MiB processed) in 7.77s]
23:34:14  9 of 11 OK created sql table model tiffany_staging.int_ntd__monthly_ridership_with_adjustments_vrh  [CREATE TABLE (622.2k rows, 2.9 MiB processed) in 7.77s]
23:34:14  7 of 11 OK created sql table model tiffany_staging.int_ntd__monthly_ridership_with_adjustments_voms  [CREATE TABLE (622.2k rows, 2.9 MiB processed) in 8.54s]
23:34:14  10 of 11 START sql table model tiffany_staging.int_ntd__monthly_ridership_with_adjustments_joined  [RUN]
23:34:21  10 of 11 OK created sql table model tiffany_staging.int_ntd__monthly_ridership_with_adjustments_joined  [CREATE TABLE (622.2k rows, 265.7 MiB processed) in 6.90s]
23:34:21  11 of 11 START sql table model tiffany_mart_ntd.dim_monthly_ntd_ridership_with_adjustments  [RUN]
23:34:27  11 of 11 OK created sql table model tiffany_mart_ntd.dim_monthly_ntd_ridership_with_adjustments  [CREATE TABLE (622.2k rows, 96.2 MiB processed) in 6.00s]
23:34:27  
23:34:27  Finished running 11 table models in 0 hours 0 minutes and 29.15 seconds (29.15s).
23:34:27  
23:34:27  Completed successfully
23:34:27  
23:34:27  Done. PASS=11 WARN=0 ERROR=0 SKIP=0 TOTAL=11
jovyan@jupyter-tiffanychu90 ~/data-infra/warehouse (ntd-doc-macros) $ poetry run dbt run -s +fct_annual_service_modes
23:34:40  Running with dbt=1.5.1
23:34:42  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
23:34:43  Found 422 models, 973 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
23:34:43  
23:34:47  Concurrency: 8 threads (target='dev')
23:34:47  
23:34:47  1 of 2 START sql view model tiffany_staging.stg_ntd_annual_data__2022__service_by_mode  [RUN]
23:34:48  1 of 2 OK created sql view model tiffany_staging.stg_ntd_annual_data__2022__service_by_mode  [CREATE VIEW (0 processed) in 0.95s]
23:34:48  2 of 2 START sql table model tiffany_mart_ntd.fct_annual_service_modes ......... [RUN]
23:34:52  2 of 2 OK created sql table model tiffany_mart_ntd.fct_annual_service_modes .... [CREATE TABLE (3.7k rows, 16.2 MiB processed) in 3.42s]
23:34:52  
23:34:52  Finished running 1 view model, 1 table model in 0 hours 0 minutes and 8.26 seconds (8.26s).
23:34:52  
23:34:52  Completed successfully
23:34:52  
23:34:52  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2
jovyan@jupyter-tiffanychu90 ~/data-infra/warehouse (ntd-doc-macros) $ poetry run dbt docs generate
23:35:04  Running with dbt=1.5.1
23:35:07  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
23:35:07  Found 422 models, 973 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
23:35:07  
23:35:16  Concurrency: 8 threads (target='dev')
23:35:16  
23:35:58  Building catalog
23:36:21  Catalog written to /home/jovyan/data-infra/warehouse/target/catalog.json

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

• No action required
• Actions required (specified below)

[github-actions]

Warehouse report 📦

DAG

Legend (in order of precedence)

Resource type	Indicator	Resolution
Large table-materialized model	Orange	Make the model incremental
Large model without partitioning or clustering	Orange	Add partitioning and/or clustering
View with more than one child	Yellow	Materialize as a table or incremental
Incremental	Light green
Table	Green
View	White

[vevetron]

Looks great!