Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accelerated compaction support #1359

Closed
m09526 opened this issue Sep 22, 2023 · 0 comments · Fixed by #1360
Closed

Accelerated compaction support #1359

m09526 opened this issue Sep 22, 2023 · 0 comments · Fixed by #1360
Labels
enhancement New feature or request
Milestone

Comments

@m09526
Copy link
Member

m09526 commented Sep 22, 2023

Background
Data compactions make up a significant part of the background computational (and therefore financial) cost of Sleeper. The native implementation in Java may be accelerated by writing the compaction algorithm in a different language. We want to experiment with alternative accelerated based compaction to test this. This maybe acceleration through HW such as GPU support or through alternative software implementations.

The compactions must be compatible with the Java version, e.g. updating the state stores and file statuses of compacted files, producing quantiles sketches of the data etc.

Description
Have a configurable table property which allows for compactions to use a different method for compaction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants