Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[spark] Compaction add parallelize parallelism to avoid small partitions #4158

Merged
merged 4 commits into from
Sep 13, 2024
Merged

[spark] Compaction add parallelize parallelism to avoid small partitions #4158

merged 4 commits into from
Sep 13, 2024

Conversation

askwang
Copy link
Contributor

@askwang askwang commented Sep 10, 2024

Purpose

Linked issue: #4157

Tests

API and Format

Documentation

@askwang
Copy link
Contributor Author

askwang commented Sep 10, 2024

can you have time to cc @Zouxxyy

@xuzifu666
Copy link
Contributor

xuzifu666 commented Sep 10, 2024 •
edited
Loading

  1. unaware-bucket table also not support parallelism（CompactProcedure##compactUnAwareBucketTable），should also support it
  2. if 1 is ok, maybe can use a unit config？

@askwang askwang changed the title [spark] Add read parallelism config for aware-bucket compaction [spark] Compaction add parallelize parallelism to avoid small partitions Sep 12, 2024
@JingsongLi
Copy link
Contributor

+1

@JingsongLi JingsongLi merged commit 2c45ac0 into apache:master Sep 13, 2024
10 checks passed
@askwang askwang mentioned this pull request Sep 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@askwang @xuzifu666 @JingsongLi