ADAP-1132: Migrate dbt-spark into dbt-adapters #404

mikealfare · 2025-01-13T17:50:10Z

Migrate dbt-spark into this repo using the following shell script and call:

repo=$1
source_branch=$2
target_branch=$3

# create a remote for the legacy adapter repo and fetch the latest commits
git remote remove old || true
git remote add old https://github.com/dbt-labs/$repo.git
git fetch old

# merge the updated branch from the legacy repo into the dbt-adapters repo
git checkout $target_branch
git merge old/$source_branch --allow-unrelated-histories

# remove the remote that was created by this process
git remote remove old || true

source ./scripts/migrate-adapter.sh dbt-spark monorepo/stage monorepo/migrate-dbt-spark

Update workflows to include dbt-spark.

Make updates related with code quality checks due to slightly different config.

Merge this PR as a regular merge instead of a squash merge in order to retain git blame history.

* Fix password is not passing for thrift + LDAP. * Add password to tests. * Add CHANGELOG entry. * Fixing up changelog entry Co-authored-by: leahwicz <[email protected]> Co-authored-by: Leah Antkiewicz <[email protected]>

* Supports new error messages. * changie * tuple * fix. * suggestion.

* updating tox python version, changing contriburing file example for integration tests * remove change to contributing file for backport ease * add changelog * trying to incorprorate mulit python version * trying to fix unit test * remove unneded call to basepython * back to 3.8

* add mypy ignore to column, connections and init * passenv spacing changes post tox release of 4.0 * adding allowlist_externals to trigger bash commands * add default_factory lambda * add extra allowlist external * add changie * experiment with default_policy Co-authored-by: Matthew McKnight <[email protected]>

* incremental predicate support * remove logging statement * test merge predicates * missing color column * add update row * changie * pin core branch * update test to remove incremental strategy * add functional test * add file format config to merge test * add plus to config name * update SQL to spark syntax * add pytest decorator * update skip profile pytest decorator * handle predicates config, add test * unpin feature request

* add release workflow * tweak the release * update needs

Co-authored-by: Github Build Bot <[email protected]>

* add mypy ignore to column, connections and init * skip databricks integration tests

* add mypy ignore to column, connections and init * add databricks integ tests back * remove exclusion * fix pydantic version issue

* Use CircleCI Context for AWS access key environment variables * Use the `aws-credentials` global context configured within CircleCI * Remove changelog entry [skip ci] * Remove extra space

* add escape_single_quotes macro * added changelog entry * Trim trailing whitespace * Trim trailing whitespace Co-authored-by: Doug Beatty <[email protected]>

* rename exceptions * point back to main * Delete Under the Hood-20230110-105904.yaml * Update Under the Hood-20230110-101919.yaml

* add mypy ignore to column, connections and init * wrap if expression in brackets * change comparison type for boolean if check * align artifact downloads

* Bumping version to 1.5.0a1 and generate CHANGELOG * update changelog for 1.5.0a1 Co-authored-by: Github Build Bot <[email protected]> Co-authored-by: Emily Rockman <[email protected]>

* Convert incremental on_schema_change tests * Switch to dbt-core main

* add mypy ignore to column, connections and init * change from agate to AttrDict

* allow thrift 0.16.0 * add changie

* Update release workflow * Fix format * Set default `test_run` value to `true` * Update Slack secret * Resolve review comments

…elog (#603)

- Update AWS secrets - Rework condition for Slack notification

* change type to string * reduce required length to 1

* Begin conversion and get ready for CI testing. * Uncheck models * Change core index. * Pair test down for minimal test * Adjust code with some guessing. * Forgot a version tag * Make test conversion work. Finally * Fix up the code. * Attempt to fix test conversion with profile skips. * Add missing column test and cleanup code. * Remove shas from the requirements now that base conversion is live. * Revert whitespace change. --------- Co-authored-by: Mila Page <[email protected]>

* update regex for version bump * update to prekind * more renaming * finish up adding nightly release check for future proofing

mikealfare · 2025-01-13T18:19:47Z

Integration tests run against this branch: https://github.com/dbt-labs/dbt-adapters/actions/runs/12753297904. These don't run in the PR checks because it's using the version on main. The attached run was run using the workflows on the PR branch and the code on the PR branch.

dbt-spark/CHANGELOG.md

dbt-spark/src/dbt/include/spark/dbt_project.yml

.pre-commit-config.yaml

VersusFacit

soft approve with comments. I assume we're fine

VShkaberda and others added 30 commits October 11, 2022 14:56

Fix/ldap password (#396)

37dcfe3

* Fix password is not passing for thrift + LDAP. * Add password to tests. * Add CHANGELOG entry. * Fixing up changelog entry Co-authored-by: leahwicz <[email protected]> Co-authored-by: Leah Antkiewicz <[email protected]>

[CT-1469] init pr for short term python version issue fix (#508)

b759267

dbt run fails with spark-testing profile of examples (#513)

c87b6b2

fix flake8 change (#516)

b6cee1d

Supports new error messages. (#520)

becc101

* Supports new error messages. * changie * tuple * fix. * suggestion.

add release workflow (#550)

6a003fb

* add release workflow * tweak the release * update needs

fix footer, update contributors (#553)

57e4ae8

Bumping version to 1.4.0b1 and generate CHANGELOG (#554)

d4c19cc

Co-authored-by: Github Build Bot <[email protected]>

add pypi release to workflow (#555)

512b3d0

skip databricks integration tests (#575)

764cea7

* add mypy ignore to column, connections and init * skip databricks integration tests

add databricks integ tests back (#578)

5ca20be

* add mypy ignore to column, connections and init * add databricks integ tests back * remove exclusion * fix pydantic version issue

Use CircleCI Context for AWS access key environment variables (#582)

d9a3d76

* Use CircleCI Context for AWS access key environment variables * Use the `aws-credentials` global context configured within CircleCI * Remove changelog entry [skip ci] * Remove extra space

add escape_single_quotes macro (#573)

28a9d09

* add escape_single_quotes macro * added changelog entry * Trim trailing whitespace * Trim trailing whitespace Co-authored-by: Doug Beatty <[email protected]>

rename exceptions (#585)

94148f6

* rename exceptions * point back to main * Delete Under the Hood-20230110-105904.yaml * Update Under the Hood-20230110-101919.yaml

wrap if expression in brackets (#576)

8d266fd

* add mypy ignore to column, connections and init * wrap if expression in brackets * change comparison type for boolean if check * align artifact downloads

Bumping version to 1.5.0a1 and generate changelog (#588)

8aceeb4

* Bumping version to 1.5.0a1 and generate CHANGELOG * update changelog for 1.5.0a1 Co-authored-by: Github Build Bot <[email protected]> Co-authored-by: Emily Rockman <[email protected]>

make dev installs requirements.txt too (#399)

89f0cc2

Convert incremental on_schema_change tests (#593)

39800e0

* Convert incremental on_schema_change tests * Switch to dbt-core main

Use AttrDict instead of agate (#604)

3114331

* add mypy ignore to column, connections and init * change from agate to AttrDict

Upgrade Thrift to 0.16.0 (#605)

2e767bc

* allow thrift 0.16.0 * add changie

[CI/CD] Update release workflow (#592)

7d46e18

* Update release workflow * Fix format * Set default `test_run` value to `true` * Update Slack secret * Resolve review comments

remove PR requirement for changie and allow multiple issues per chang…

4517567

…elog (#603)

Update release workflow (#613)

4d179e0

- Update AWS secrets - Rework condition for Slack notification

change type to string (#611)

9241c38

* change type to string * reduce required length to 1

fix bug with contributor list (#620)

5d27961

update regex for version bump (#630)

55d86d0

* update regex for version bump * update to prekind * more renaming * finish up adding nightly release check for future proofing

add dbt-spark as option for testing integration tests workflow

39cabd9

mikealfare temporarily deployed to dbt-spark January 13, 2025 18:05 — with GitHub Actions Inactive

mikealfare temporarily deployed to dbt-spark January 13, 2025 18:06 — with GitHub Actions Inactive

mikealfare had a problem deploying to dbt-spark January 13, 2025 18:06 — with GitHub Actions Failure

mikealfare had a problem deploying to dbt-athena January 13, 2025 18:06 — with GitHub Actions Failure

mikealfare temporarily deployed to dbt-athena January 13, 2025 18:06 — with GitHub Actions Inactive

mikealfare marked this pull request as ready for review January 13, 2025 18:19

mikealfare requested a review from a team as a code owner January 13, 2025 18:19

mikealfare temporarily deployed to dbt-spark January 13, 2025 19:45 — with GitHub Actions Inactive

mikealfare temporarily deployed to dbt-athena January 13, 2025 19:51 — with GitHub Actions Inactive

mikealfare had a problem deploying to dbt-athena January 13, 2025 20:33 — with GitHub Actions Failure

mikealfare temporarily deployed to dbt-athena January 13, 2025 20:33 — with GitHub Actions Inactive

mikealfare temporarily deployed to dbt-athena January 13, 2025 21:50 — with GitHub Actions Inactive

VersusFacit reviewed Jan 13, 2025

View reviewed changes

dbt-spark/CHANGELOG.md Show resolved Hide resolved

VersusFacit reviewed Jan 13, 2025

View reviewed changes

dbt-spark/src/dbt/include/spark/dbt_project.yml Show resolved Hide resolved

VersusFacit reviewed Jan 13, 2025

View reviewed changes

.pre-commit-config.yaml Show resolved Hide resolved

VersusFacit approved these changes Jan 13, 2025

View reviewed changes

mikealfare merged commit 5983011 into main Jan 13, 2025
79 checks passed

mikealfare deleted the monorepo/migrate-dbt-spark branch January 13, 2025 22:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADAP-1132: Migrate dbt-spark into dbt-adapters #404

ADAP-1132: Migrate dbt-spark into dbt-adapters #404

mikealfare commented Jan 13, 2025

mikealfare commented Jan 13, 2025

VersusFacit left a comment

ADAP-1132: Migrate dbt-spark into dbt-adapters #404

ADAP-1132: Migrate dbt-spark into dbt-adapters #404

Conversation

mikealfare commented Jan 13, 2025

mikealfare commented Jan 13, 2025

VersusFacit left a comment

Choose a reason for hiding this comment