Add kwargs param in DocsOperator method upload_to_cloud_storage
#1422
+12
−5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
Deploying astronomer-cosmos with
|
| Latest commit: |
ceead6e
|
| Status: | ⚡️ Build in progress... |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1422 +/- ##
=======================================
Coverage 96.52% 96.52%
=======================================
Files 73 73
Lines 4320 4320
=======================================
Hits 4170 4170
Misses 150 150 ☔ View full report in Codecov by Sentry. |
dosubot
bot
added
the
size:S
pankajkoti
approved these changes
Dec 26, 2024
There was a problem hiding this comment.
Thanks for the quick fix @pankajastro . Indeed, it appears that the example DAG we have was silently not running the tasks as it could not find the connections. I have created a PR #1428 to adapt to our existing CI connections and run this example DAG and as can be seen in the action https://github.com/astronomer/astronomer-cosmos/actions/runs/12500718837/job/34877247783?pr=1428 it failed with the same reason as in #1420.
I will rebase my PR after merging this one to verify CI passes after running the docs DAG.
tatiana
pushed a commit
that referenced
this pull request
Dec 27, 2024
It seems that the dbt docs DAG in our example DAGs suite was silently failing to run because the specified connections in the DAG were not present in our CI environment. I am removing the connection checks, allowing the previously skipped tasks to execute even when the remote Airflow connections are unavailable. Additionally, I’ve updated the DAG to use our existing CI credentials and Airflow connections, as well as updated the bucket name to the ones I created in our object stores. This change will help catch issues like #1420 earlier. While fix #1422 has already addressed #1420, this PR will now validate that the fix is functioning as expected. closes: #1420
Merged
pankajastro
added a commit
that referenced
this pull request
Dec 30, 2024
1.8.1 (2024-12-30) -------------------- Bug Fixes * Fix rendering dbt tests with multiple parents by @tatiana in #1433 * Add ``kwargs`` param in DocsOperator method ``upload_to_cloud_storage`` by @pankajastro in #1422 Docs * Improve OpenLineage documentation by @tatiana in #1431 Others * Enable Docs DAG in CI leveraging existing CI connections by @pankajkoti in #1428 * Install providers with airflow by @pankajkoti in #1432 * Remove unused docs dependency by @pankajastro in #1414 * Pre-commit hook updates in #1424 --------- Co-authored-by: Tatiana Al-Chueyr <[email protected]> Co-authored-by: Pankaj Koti <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses an issue where the
callbackcallable parameter'shandling was modified in PR #1389 leading to a change in
the way keyword arguments like context were passed to the callable.
However, the Docs Operator has its own implementation of the callback callable,
which only expects a single parameter, project_dir. As a result, passing extra
keyword arguments like context is causing a mismatch and resulting in the error described in #1420.
This PR add kwargs param in upload_to_cloud_storage for Docs operators
We have integration tests for these operator but look like CI does have not required setup and it get ignored.
https://github.com/astronomer/astronomer-cosmos/blob/main/dev/dags/dbt_docs.py
related: #1420