archival: Use read-write fence in the ntp_archiver #24574
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
2e39592 to
a392416
Compare
Retry command for Build#59774please wait until all jobs are finished before running the slash command |
CI test resultstest results on build#59774
test results on build#59783
test results on build#59824
test results on build#59868
test results on build#59928
test results on build#59958
test results on build#59971
test results on build#59988
test results on build#60029
test results on build#60039
test results on build#60048
test results on build#60128
test results on build#60147
|
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
a392416 to
ce0cb32
Compare
Retry command for Build#59783please wait until all jobs are finished before running the slash command |
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
ce0cb32 to
06ca1b0
Compare
Retry command for Build#59824please wait until all jobs are finished before running the slash command |
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
06ca1b0 to
63eb5f5
Compare
Retry command for Build#59868please wait until all jobs are finished before running the slash command |
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
63eb5f5 to
b7836dd
Compare
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
b7836dd to
a52b914
Compare
Retry command for Build#59928please wait until all jobs are finished before running the slash command |
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
a52b914 to
9cb55f3
Compare
Retry command for Build#59958please wait until all jobs are finished before running the slash command |
|
/ci-repeat 1 |
Retry command for Build#59988please wait until all jobs are finished before running the slash command |
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
2 times, most recently
from
83a0abc to
8b33548
Compare
Retry command for Build#60029please wait until all jobs are finished before running the slash command |
|
/ci-repeat 1 |
Retry command for Build#60039please wait until all jobs are finished before running the slash command |
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
ba45e29 to
9b0a748
Compare
Retry command for Build#60048please wait until all jobs are finished before running the slash command |
|
/ci-repeat 1 |
The snapshot doesn't serialize the applied-offset field of the manifest. This commit fixes this. Signed-off-by: Evgeny Lazin <[email protected]>
The fence is a value which is produced by recording the current offset of the last applied archival STM command. The fence is added to the archival STM config batch. If no commands were applied to the STM the offset of the last applied command will be the same and the configuration batch will be applied. Otherwise, if some command was applied after the record batch was constructed the batch will not be applied. This is essentially a concurrency control mechamism. The commands are already present in the STM but not used as for now. The existing metadata validation mechanism stays.
The check is no longer needed because we're using fencing mechanism to prevent concurrency. The check is actually performed when the segment is uploaded. The removed check was used as an optimistic concurrency control mechanism and is now replaced by the RW-fence. Also, deprecate the error code that STM uses to communicate the consistency violation to the caller. The error code is not used anywhere else. Signed-off-by: Evgeny Lazin <[email protected]>
The option was never turned on by default. Our plan was to eventually enable it but it is now replaced with fencing mechanism which is always on and can't be disabled. Signed-off-by: Evgeny Lazin <[email protected]>
This commit enables rw-fence mechanism for housekeeping reuploads. In case if the housekeeping is running on a stale version of the STM snapshot the update will be rejected. Signed-off-by: Evgeny Lazin <[email protected]>
Use rw-fence for both normal retention/GC and spillover. Currently, the upload metadata validation only checks added segments. Because of that there is still some room for concurrent updates during the housekeeping (GC or retention). With rw-fence the metadata updates related to retention or GC will become safer. Signed-off-by: Evgeny Lazin <[email protected]>
Signed-off-by: Evgeny Lazin <[email protected]>
Previous implementation of the archival STM had a bug because of which the last applied offset wasn't added to the snapshot. This could potentially make replicas diverge. The solution is to add a feature flag and use the rw-fence mechanism only if all replicas are upgraded. Signed-off-by: Evgeny Lazin <[email protected]>
Signed-off-by: Evgeny Lazin <[email protected]>
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
9b0a748 to
779bc98
Compare
Split method into two parts: - wait_uploads_complete - replicate_archival_metadata The first method is waiting until all uploads are finished. But unlike wait_uploads it doesn't replicate the metadata. Instead of doing this it just returns all information needed to replicate it. The second method takes the output of the wait_uploads_complete and replicates it. Finally, the wait_uploads method is now implemented using this two new methods. This is a preparation for the next commit. Currently, when we're uploading both compacted and non-compacted segments in the same iteration of the upload loop they both replicated independently. This worked fine before (with the exception of the race condition on _projected_manifest_clean_at). But with the fencing mechanism one of this wait_uploads calls will fail to replicate because of the fence. Currently, the same fence is used to invoke wait_uploads for compacted and non-compacted segments. The solution to this problem is to use wait_uploads_complete to wait until both compacted and non-compacted uploads are completed and then use a single replicate_archival_metadata call. This call will be able to use the fence value correctly. Signed-off-by: Evgeny Lazin <[email protected]>
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
c52583f to
aa6837a
Compare
Put non-compacted segment metadata and compacted segment metadata into the same archival control batch. This is needed to make fencing work correctly if both comapcted and non-compacted uploads are made in the same upload loop iteraton. Signed-off-by: Evgeny Lazin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The fence is a value which is produced by recording the current offset of the last applied archival STM command. The fence is added to the archival STM config batch. If no commands were applied to the STM the offset of the last applied command will be the same and the configuration batch will be applied. Otherwise, if some command was applied after the record batch was constructed the batch will not be applied.
This is essentially a concurrency control mechanism. The commands are already present in the STM but not used as for now. The existing metadata validation mechanism in the NTP archiver is still working but we no longer need to check the commands in the STM.
Backports Required
Release Notes