Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data test broken when switching to materialized views on postgres #1684

Closed
mateusz opened this issue Aug 27, 2024 · 2 comments
Closed

Data test broken when switching to materialized views on postgres #1684

mateusz opened this issue Aug 27, 2024 · 2 comments
Labels
Bug Something isn't working Triage 👀

Comments

@mateusz
Copy link

mateusz commented Aug 27, 2024

Describe the bug

On postgres, when using elementary.column_anomalies test on a column that is part of a table using materialized views, an "Unable to find column" error occurs during the elemenatary data test run.

To Reproduce

Create a table using materialized view materialization:

{{ config(materialized='materialized_view') }}

SELECT bill_usd FROM table

Configure test in DBT table spec:

models:
  - name: table
    columns:
      - name: bill_usd
        tests:
          - elementary.column_anomalies:
              detection_period:
                period: day
                count: 2
              training_period:
                period: day
                count: 35
              detection_delay:
                period: day
                count: 5
              column_anomalies:
                - sum
              config:
                severity: warn

Run elementary:

2024-08-27 12:05:46,796 | INFO | root | 00:05:42    Compilation Error in test ...bill_usd__day__5__day__2__day__35 (models/....yml)
2024-08-27 12:05:46,796 | INFO | root |   Unable to find column `bill_usd` in `....TABLE`
2024-08-27 12:05:46,796 | INFO | root |   
2024-08-27 12:05:46,796 | INFO | root |   > in macro test_column_anomalies (macros/edr/tests/test_column_anomalies.sql)
2024-08-27 12:05:46,797 | INFO | root |   > called by test ...bill_usd__day__5__day__2__day__35 (...)

This problem goes away when I remove the materialization, or switch to table materialization.

Expected behavior

I expect data tests to continue working when I switch to materialized_view materialization.

Screenshots

n/a

Environment (please complete the following information):

  • Elementary CLI (edr) version: 0.16.1
  • Elementary dbt package version: 0.16.0
  • dbt version you're using: 1.8.5
  • Data warehouse: postgres
  • Infrastructure details (e.g. operating system, prod / dev / staging, deployment infra, CI system, etc): freshly built docker image prefecthq/prefect:2-python3.9
@mateusz mateusz added Bug Something isn't working Triage 👀 labels Aug 27, 2024
@mateusz mateusz changed the title Data test broken when switching to materialized views materialization on postgres Data test broken when switching to materialized views on postgres Aug 27, 2024
@ofek1weiss
Copy link
Contributor

Hey @mateusz
This issue seems to be caused by the postgres dbt adapter itself... 😞
Elementary uses the macro adapter.get_columns_in_relation, to find metadata about the monitored column, and the adapter itself returns an empty list for materialized views.
I recommend opening an issue in the dbt-postgres repo for this issue.

@mateusz
Copy link
Author

mateusz commented Sep 23, 2024

Thanks for looking, will do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Triage 👀
Projects
None yet
Development

No branches or pull requests

4 participants
@mateusz @ofek1weiss and others