Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add missing column "key" to fct_daily_rt_feed_validation_notices #3510

Merged
merged 1 commit into from
Oct 19, 2024

Conversation

erikamov
Copy link
Contributor

Description

Optimization done on 'fct_daily_rt_feed_validation_notices' for issue #3481 worked and DAG is passing, but I noticed errors on tests while checking the logs:

Screenshot 2024-10-19 at 10 12 49 AM

So I run tests locally and noticed we were missing one column "key":

Screenshot 2024-10-19 at 10 13 06 AM

To fix I simply added the column back and the tests passed:

Screenshot 2024-10-19 at 10 19 57 AM

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Tested locally and table created on Staging:

❯ poetry run dbt run -s "models/mart/gtfs_quality/fct_daily_rt_feed_validation_notices.sql"
17:09:42  Running with dbt=1.5.1
17:09:44  Found 420 models, 949 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
17:09:44
17:09:46  Concurrency: 8 threads (target='dev')
17:09:46
17:09:46  1 of 1 START sql table model erika_mart_gtfs_quality.fct_daily_rt_feed_validation_notices  [RUN]
17:09:55  1 of 1 OK created sql table model erika_mart_gtfs_quality.fct_daily_rt_feed_validation_notices  [CREATE TABLE (206.2k rows, 39.2 GiB processed) in 8.82s]
17:09:55
17:09:55  Finished running 1 table model in 0 hours 0 minutes and 10.97 seconds (10.97s).
17:09:55
17:09:55  Completed successfully
17:09:55
17:09:55  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
 ❯ poetry run dbt test -s "models/mart/gtfs_quality/fct_daily_rt_feed_validation_notices.sql"
17:10:00  Running with dbt=1.5.1
17:10:01  Found 420 models, 949 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 175 sources, 4 exposures, 0 metrics, 0 groups
17:10:01
17:10:02  Concurrency: 8 threads (target='dev')
17:10:02
17:10:02  1 of 15 START test dbt_utils_expression_is_true_fct_daily_rt_feed_validation_notices_parsed_files___4320  [RUN]
17:10:02  2 of 15 START test dbt_utils_expression_is_true_fct_daily_rt_feed_validation_notices_validated_extracts___4320  [RUN]
17:10:02  3 of 15 START test dbt_utils_expression_is_true_fct_daily_rt_feed_validation_notices_validated_extracts_validation_successes_validation_exceptions  [RUN]
17:10:02  4 of 15 START test dbt_utils_expression_is_true_fct_daily_rt_feed_validation_notices_validation_exceptions___4320  [RUN]
17:10:02  5 of 15 START test dbt_utils_expression_is_true_fct_daily_rt_feed_validation_notices_validation_successes___4320  [RUN]
17:10:02  6 of 15 START test dbt_utils_not_null_proportion_fct_daily_rt_feed_validation_notices_0_99__parsed_files  [RUN]
17:10:02  7 of 15 START test dbt_utils_not_null_proportion_fct_daily_rt_feed_validation_notices_0_99__validated_extracts  [RUN]
17:10:02  8 of 15 START test dbt_utils_not_null_proportion_fct_daily_rt_feed_validation_notices_0_99__validation_exceptions  [RUN]
17:10:03  1 of 15 PASS dbt_utils_expression_is_true_fct_daily_rt_feed_validation_notices_parsed_files___4320  [PASS in 1.12s]
17:10:03  9 of 15 START test dbt_utils_not_null_proportion_fct_daily_rt_feed_validation_notices_0_99__validation_successes  [RUN]
17:10:03  3 of 15 PASS dbt_utils_expression_is_true_fct_daily_rt_feed_validation_notices_validated_extracts_validation_successes_validation_exceptions  [PASS in 1.14s]
17:10:03  10 of 15 START test not_null_fct_daily_rt_feed_validation_notices_code ......... [RUN]
17:10:03  7 of 15 PASS dbt_utils_not_null_proportion_fct_daily_rt_feed_validation_notices_0_99__validated_extracts  [PASS in 1.24s]
17:10:03  2 of 15 PASS dbt_utils_expression_is_true_fct_daily_rt_feed_validation_notices_validated_extracts___4320  [PASS in 1.24s]
17:10:03  11 of 15 START test not_null_fct_daily_rt_feed_validation_notices_date ......... [RUN]
17:10:03  12 of 15 START test not_null_fct_daily_rt_feed_validation_notices_description .. [RUN]
17:10:03  5 of 15 PASS dbt_utils_expression_is_true_fct_daily_rt_feed_validation_notices_validation_successes___4320  [PASS in 1.28s]
17:10:03  13 of 15 START test not_null_fct_daily_rt_feed_validation_notices_is_critical .. [RUN]
17:10:03  6 of 15 PASS dbt_utils_not_null_proportion_fct_daily_rt_feed_validation_notices_0_99__parsed_files  [PASS in 1.35s]
17:10:03  14 of 15 START test not_null_fct_daily_rt_feed_validation_notices_key .......... [RUN]
17:10:04  4 of 15 PASS dbt_utils_expression_is_true_fct_daily_rt_feed_validation_notices_validation_exceptions___4320  [PASS in 1.53s]
17:10:04  15 of 15 START test unique_fct_daily_rt_feed_validation_notices_key ............ [RUN]
17:10:04  8 of 15 PASS dbt_utils_not_null_proportion_fct_daily_rt_feed_validation_notices_0_99__validation_exceptions  [PASS in 1.67s]
17:10:04  9 of 15 PASS dbt_utils_not_null_proportion_fct_daily_rt_feed_validation_notices_0_99__validation_successes  [PASS in 0.93s]
17:10:04  12 of 15 PASS not_null_fct_daily_rt_feed_validation_notices_description ........ [PASS in 0.88s]
17:10:04  11 of 15 PASS not_null_fct_daily_rt_feed_validation_notices_date ............... [PASS in 0.90s]
17:10:04  14 of 15 PASS not_null_fct_daily_rt_feed_validation_notices_key ................ [PASS in 0.89s]
17:10:04  10 of 15 PASS not_null_fct_daily_rt_feed_validation_notices_code ............... [PASS in 1.20s]
17:10:05  13 of 15 PASS not_null_fct_daily_rt_feed_validation_notices_is_critical ........ [PASS in 1.23s]
17:10:05  15 of 15 PASS unique_fct_daily_rt_feed_validation_notices_key .................. [PASS in 1.13s]
17:10:05
17:10:05  Finished running 15 tests in 0 hours 0 minutes and 3.93 seconds (3.93s).
17:10:05
17:10:05  Completed successfully
17:10:05
17:10:05  Done. PASS=15 WARN=0 ERROR=0 SKIP=0 TOTAL=15

Post-merge follow-ups

  • No action required
  • Actions required (specified below)

Watch the next time the DAG runs to see the results and no more errors.

@erikamov erikamov self-assigned this Oct 19, 2024
@erikamov erikamov requested a review from ohrite October 19, 2024 17:22
@erikamov erikamov changed the title Add missing key column to fct_daily_rt_feed_validation_notices Add missing column "key" to fct_daily_rt_feed_validation_notices Oct 19, 2024
Copy link

Warehouse report 📦

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

@erikamov erikamov merged commit aff55ca into main Oct 19, 2024
4 checks passed
@erikamov erikamov deleted the 3481-validation-notices branch October 19, 2024 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants