Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore invalid compaction job commits in a transaction #3900

Open
rtjd6554 opened this issue Dec 10, 2024 · 0 comments
Open

Ignore invalid compaction job commits in a transaction #3900

rtjd6554 opened this issue Dec 10, 2024 · 0 comments
Milestone

Comments

@rtjd6554
Copy link
Collaborator

Background

Split from:

Description

We'll need to adjust how compaction commit transactions are applied so that any invalid commits are discarded, and the transaction as a whole is still applied, so that if a job is run more than once we ignore the second run. If this results in an empty transaction we can just not add it to the log.

Analysis

Compaction commits could be invalid for a various reasons:
If input files no longer exists, (likely due to already have been committed earlier) that commit can be ignored,
If input files exists but aren't assigned to job, want to log the event out but all for the compactions to continue (i.e. not an exception)

@patchwork01 patchwork01 changed the title Compaction Commit Job logic adjustment Compaction job commit logic adjustment Dec 10, 2024
@patchwork01 patchwork01 changed the title Compaction job commit logic adjustment Ignore invalid compaction job commits in a transaction Dec 10, 2024
@patchwork01 patchwork01 added this to the 0.28.0 milestone Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants