Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Add missing merge clause types in DeltaTableWriter #43

Closed
riccamini opened this issue Jun 7, 2024 · 0 comments · Fixed by #46
Closed

[FEATURE] Add missing merge clause types in DeltaTableWriter #43

riccamini opened this issue Jun 7, 2024 · 0 comments · Fixed by #46
Labels
enhancement New feature or request

Comments

@riccamini
Copy link
Contributor

Is your feature request related to a problem? Please describe.

DeltaTableWriter provides the option to configure the writer in MERGE mode using the output_mode_params field by providing in it a list of merge clauses in the merge_builder key. This is especially useful in case the Delta table or the DataFrame to be merged are not available upfront. Not all merge clauses are currently supported (whenMatchedUpdateAll and whenNotMatchedInsertAll are not covered).

Describe the solution you'd like

It should be possible to use all clauses in the configuration

Describe alternatives you've considered

No alternative solutions can be used in this case because this is the only way of configuring the writer in MERGE mode when the table and the DataFrame are not available when we create the DeltaTableWriter

Additional context

...
@riccamini riccamini added the enhancement New feature or request label Jun 7, 2024
@riccamini riccamini mentioned this issue Jun 7, 2024
Closed
9 tasks
@dannymeijer dannymeijer linked a pull request Jun 10, 2024 that will close this issue
Add missing merge clauses in DeltaTableWriter #44
Closed
9 tasks
@riccamini riccamini mentioned this issue Jun 10, 2024
Merged
9 tasks
dannymeijer pushed a commit that referenced this issue Jun 10, 2024
@riccamini
Add missing merge clauses in DeltaTableWriter (#46)
4c4d062 
DeltaTableWriter provides the option to configure the writer in MERGE
mode using the output_mode_params field by providing in it a list of
merge clauses in the merge_builder key. This is especially useful in
case the Delta table or the DataFrame to be merged are not available
upfront. Not all merge clauses were supported: the merge clause provided
was being validated against a list of valid clauses which did not
include whenMatchedUpdateAll and whenNotMatchedInsertAll clauses. As
part of this PR:

- validation of the clauses now happens in the output_mode_params field
validator (currently this is happening only when the merge is being
executed)
- the list of available merge clauses is sourced directly from available
methods in DeltaMergeBuilder class and now includes all of them

## Related Issue
#43

## Motivation and Context
Now it's possible to use all merge clauses when configuring the
DeltaTableWriter

---------

Co-authored-by: riacom_nike <[email protected]>
@dannymeijer dannymeijer linked a pull request Jun 10, 2024 that will close this issue
Add missing merge clauses in DeltaTableWriter #46
Merged
9 tasks
@dannymeijer dannymeijer moved this to Done in Koheesio Jun 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
Koheesio
Status: Done
Milestone
No milestone
2 participants
@dannymeijer @riccamini