parse + compile constraint.to and constraint.to_columns on foreign key constraints

[MichelleArk]

resolves #8062
associated dbt-adapters PR: dbt-labs/dbt-adapters#260

Problem

Because you must hard-code your database.schema.table name when setting a foreign key constraint:

• DAG dependencies are incorrect
• multi-environment is not supported (it's very hacky)
• This feature has become more important now that warehouses use foreign key constraints for better performance.

Solution

During parsing:

• parse the ref or source expressions
• validate valid jinja ref/source expression
• populate node.refs and/or node.sources based on fk 'to' fields

During compilation:

• User the adapter class to render a relation, transforming the 'raw' 'to' field to a relation identifier.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
• This PR includes type annotations for new and modified functions

[github-actions]

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

[codecov]

Codecov Report

Attention: Patch coverage is 87.83784% with 9 lines in your changes missing coverage. Please review.

Project coverage is 88.86%. Comparing base (c668846) to head (dc10493).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #10414   +/-   ##
=======================================
  Coverage   88.86%   88.86%           
=======================================
  Files         180      180           
  Lines       22560    22642   +82     
=======================================
+ Hits        20047    20120   +73     
- Misses       2513     2522    +9     

Flag	Coverage Δ
integration	86.16% <87.83%> (-0.04%)	⬇️
unit	62.22% <63.51%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
Unit Tests	62.22% <63.51%> (+0.07%)	⬆️
Integration Tests	86.16% <87.83%> (-0.04%)	⬇️

[MichelleArk]

For docs:

Model-level constraint example syntax:

models:
  - name: my_model
    constraints:
      - type: foreign_key
        columns: [id]
        to: ref('my_model_to') | source('source', 'source_table')
        to_columns: [id]
    columns:
      - name: id
        data_type: integer

Column-level example syntax:

models:
  - name: my_model
    columns:
      - name: id
        data_type: integer
        constraints:
        - type: foreign_key
          to: ref('my_model_to') | source('source', 'source_table')
          to_columns: [id]

[MichelleArk]

I'd like to do the following tidying in a follow-up (to not bloat this PR scope):

• delete adapters test_constraints.py
• move primary key constraint testing to tests/constraints
• use new static jinja rendering method for unit test ref and source resolution

(will link issues shortly)

[gshank]

Looks good. Nice tests.

I do have one question -- it might be possible to do some of this at parse time. Did doing it that way get too weird? Or did it feel like it had more affinity to compilation time than parse time?

[MichelleArk]

I do have one question -- it might be possible to do some of this at parse time. Did doing it that way get too weird? Or did it feel like it had more affinity to compilation time than parse time?

That's a great question. I initially did a lot more of this at compile time but was able to move the initial syntax validation (tests) and dependency lookup (tests) to parse-time.

At compile time the only thing that happens is that the to field of constraints is rendered to a relation (link). This felt appropriate to do at compilation time as it depends on an adapter (e.g. quoting config) to render a relation.

There is error-handling in compilation for sanity's sake and to keep the internal methods generalized, but I don't expect syntax errors (e.g. invalid ref syntax) to be caught during compilation phase since parsing was run first.

[MichelleArk]

Some final 🎩
(contracted has fk constraint to my_other_model via 'to' and 'to_columns')

lineage:

❯ dbt ls --select +contracted
21:58:34  Running with dbt=1.9.0-a1
21:58:34  target not specified in profile 'postgres', using 'default'
21:58:34  Registered adapter: postgres=1.9.0-a1
21:58:35  Found 22 models, 6 seeds, 20 data tests, 6 sources, 13 metrics, 716 macros, 6 semantic models, 5 unit tests
jaffle_shop.marts.contracted
jaffle_shop.my_other_model

violating constraint:

❯ dbt run --select +contracted
21:57:00  Running with dbt=1.9.0-a1
21:57:00  target not specified in profile 'postgres', using 'default'
21:57:01  Registered adapter: postgres=1.9.0-a1
21:57:01  Found 22 models, 6 seeds, 20 data tests, 6 sources, 13 metrics, 716 macros, 6 semantic models, 5 unit tests
21:57:01  
21:57:02  Concurrency: 1 threads (target='default')
21:57:02  
21:57:02  1 of 2 START sql table model test_arky.my_other_model .......................... [RUN]
21:57:02  1 of 2 OK created sql table model test_arky.my_other_model ..................... [INSERT 0 1 in 0.21s]
21:57:02  2 of 2 START sql table model test_arky.contracted .............................. [RUN]
21:57:02  2 of 2 ERROR creating sql table model test_arky.contracted ..................... [ERROR in 0.07s]
21:57:02  
21:57:02  Finished running 2 table models in 0 hours 0 minutes and 0.60 seconds (0.60s).
21:57:02  
21:57:02  Completed with 1 error and 0 warnings:
21:57:02  
21:57:02    Database Error in model contracted (models/marts/contracted.sql)
  insert or update on table "contracted__dbt_tmp" violates foreign key constraint "contracted__dbt_tmp_id_fkey"
  DETAIL:  Key (id)=(7) is not present in table "my_other_model".
  compiled Code at target/run/jaffle_shop/models/marts/contracted.sql
21:57:02  
21:57:02  Done. PASS=1 WARN=0 ERROR=1 SKIP=0 TOTAL=2

passing constraint:

❯ dbt run --select +contracted
21:58:09  Running with dbt=1.9.0-a1
21:58:09  target not specified in profile 'postgres', using 'default'
21:58:09  Registered adapter: postgres=1.9.0-a1
21:58:10  Found 22 models, 6 seeds, 20 data tests, 6 sources, 13 metrics, 716 macros, 6 semantic models, 5 unit tests
21:58:10  
21:58:10  Concurrency: 1 threads (target='default')
21:58:10  
21:58:10  1 of 2 START sql table model test_arky.my_other_model .......................... [RUN]
21:58:10  1 of 2 OK created sql table model test_arky.my_other_model ..................... [INSERT 0 1 in 0.20s]
21:58:10  2 of 2 START sql table model test_arky.contracted .............................. [RUN]
21:58:10  2 of 2 OK created sql table model test_arky.contracted ......................... [INSERT 0 1 in 0.07s]
21:58:10  
21:58:10  Finished running 2 table models in 0 hours 0 minutes and 0.61 seconds (0.61s).
21:58:10  
21:58:10  Completed successfully
21:58:10  
21:58:10  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

expression backcompat (expression set to 'still works')

...
21:59:17  
21:59:17  Finished running 1 table model in 0 hours 0 minutes and 0.46 seconds (0.46s).
21:59:17  
21:59:17  Completed with 1 error and 0 warnings:
21:59:17  
21:59:17    Database Error in model contracted (models/marts/contracted.sql)
  syntax error at or near "works"
  LINE 14:     foreign key (id) references still works
                                                 ^
  compiled Code at target/run/jaffle_shop/models/marts/contracted.sql
21:59:17  
21:59:17  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

[ekarabulut-r7]

Thank you @MichelleArk ! with this fix, we will clear many post_hooks from our models :)

[elyobo]

This is a great addition, thanks @MichelleArk.

[caileyfitzgerald]

Is this just for dbt-core or also for dbt-cloud?

[MichelleArk]

This is implemented in dbt-core, so its definitely for dbt-core as well as dbt Cloud. It will go out in the next minor version (1.9) of dbt-core!