Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[spark] minor refactor SparkWriter #3067

Merged
merged 3 commits into from
Mar 22, 2024
Merged

[spark] minor refactor SparkWriter #3067

merged 3 commits into from
Mar 22, 2024

Conversation

YannByron
Copy link
Contributor

Purpose

Linked issue: close #xxx

Tests

API and Format

Documentation

@JingsongLi
Copy link
Contributor

Hi @YannByron , Please roughly explain why refactoring is necessary.

@YannByron
Copy link
Contributor Author

Currently, all the spark write operations use WriteIntoPaimonTable that has two write data to files and commit steps. For supporting update/delete/... on append tables, need to split these two steps: write data to files is extracted out, while commit is kept in certain write operation. As thus, like support update on append table, we can find the files that has to be rewrote, write the updated data, commit both the new data files and deleted files.

JingsongLi
JingsongLi reviewed Mar 22, 2024
View reviewed changes
...rk/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/BucketProcessor.scala Outdated

def processPartition(rowIterator: Iterator[Row]): Iterator[Row] = {
val rowType = table.rowType()
val rowKeyExtractor = table.asInstanceOf[AbstractFileStoreTable].createRowKeyExtractor()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want this extractor, you could introduce a createRowKeyExtractor to FileStoreTable.

Subsequent matters related to permission security may require us to wrap a FileStoreTable class, so it is best to rely only on the interface rather than the implementation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to do that at first...OK.

@JingsongLi
Copy link
Contributor

+1

@JingsongLi JingsongLi merged commit d4beafa into apache:master Mar 22, 2024
9 checks passed
yuzelin pushed a commit to yuzelin/paimon that referenced this pull request Mar 22, 2024
@YannByron @yuzelin
[spark] Minor refactor SparkWriter to separate commit (apache#3067)
c600f46
zhu3pang pushed a commit to zhu3pang/incubator-paimon that referenced this pull request Mar 29, 2024
@YannByron @zhu3pang
[spark] Minor refactor SparkWriter to separate commit (apache#3067)
5579dee
@YannByron
Copy link
Contributor Author

#3057

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JingsongLi JingsongLi JingsongLi left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@YannByron @JingsongLi