add support for delta lake table_changes table valued function as DBT source

[haon85]

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[codecov]

Codecov Report

Attention: Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...adata-ingestion/src/datahub/sql_parsing/_models.py	66.66%	2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (87.50%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Files with missing lines	Coverage Δ
...tion/src/datahub/sql_parsing/sql_parsing_common.py	100.00% <100.00%> (ø)
...gestion/src/datahub/sql_parsing/sqlglot_lineage.py	92.41% <100.00%> (-1.57%)	⬇️
...adata-ingestion/src/datahub/sql_parsing/_models.py	77.63% <66.66%> (+1.54%)	⬆️

... and 72 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aaaa655...9507176. Read the comment docs.

[hsheth2]

I'm not super familiar with table-valued functions in delta lake. Could you (1) link to the docs on it and (2) add some unit tests for this code

Overall - modifying the from_sqlglot_table method feels pretty hacky and intuitively doesn't feel like the right place to make these changes

[haon85]

Thanks for the review @hsheth2,
Here is the link to TVF: https://docs.databricks.com/en/sql/language-manual/functions/table_changes.html

As for how to make this change, I actually thought about it for a while and did not find a good place. The scenario is we are using delta lake table for storage and registered tables in Hive metastore, then doing ETL in DBT SQL code. To support this use case, we either make code change in sqlglot so that in the Node, TVF style table source is defined correctly, or make change when we read table source from sqlglot Node, that's what I am doing now.

I am quite new to Datahub code and can definitely use some help, if you have any suggestion to better support this use case I am all ears.

I 'll add UT soon.

[haon85]

Thanks for the review @hsheth2,
Here is the link to TVF: https://docs.databricks.com/en/sql/language-manual/functions/table_changes.html

As for how to make this change, I actually thought about it for a while and did not find a good place. The scenario is we are using delta lake table for storage and registered tables in Hive metastore, then doing ETL in DBT SQL code. To support this use case, we either make code change in sqlglot so that in the Node, TVF style table source is defined correctly, or make change when we read table source from sqlglot Node, that's what I am doing now.

I am quite new to Datahub code and can definitely use some help, if you have any suggestion to better support this use case I am all ears.

I 'll add UT soon.