Unblock group of consumers on super stream partition #7765
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
acogoluegnes
force-pushed
the
rabbitmq-server-7743-unblock-super-stream-partition-consumer-group
branch
from
1c1e70b to
90eedfa
Compare
|
I can't reproduce the issue 7743 anymore. |
acogoluegnes
force-pushed
the
rabbitmq-server-7743-unblock-super-stream-partition-consumer-group
branch
from
83e5c6c to
65c74d8
Compare
A group of consumers on a super stream can end up blocked without an active consumer. This can happen with consumer churn: one consumer gets removed, which makes the active consumer passive, but the former active consumer never gets to know because it has been removed itself. This commit changes the structure of the messages the SAC coordinator sends to consumer connections, to embed enough information to look up the group and to instruct it to choose a new active consumer when the race condition mentioned above comes up. Because of the changes in the structure of messages, a feature flag is required to make sure the SAC coordinator starts sending the new messages only when all the nodes have been upgraded. References #7743
References #7743
And squash a dialyzer warning.
The stream plugin can send frames to a client connection and expect a response from it. This is used currently for the consumer_update frame (single active consumer feature). There was no timeout mechanism so far, so a slow or blocked application could prevent a group of consumers to move on. This commit introduces a timeout mechanism: if the expected response takes too long to arrive, the server assumes the connection is blocked and closes it. The default timeout is 60 seconds but it can be changed by setting the request_timeout parameter of the rabbitmq_stream application. Note the mechanism does not enforce the exact duration of the timeout, as a timer is set for the first request and re-used for other requests. With bad timing, a request can time out after twice as long as the set-up timeout. References #7743
References #7743
…rtition-consumer-group
michaelklishin
deleted the
rabbitmq-server-7743-unblock-super-stream-partition-consumer-group
branch
michaelklishin
added a commit
that referenced
this pull request
Apr 4, 2023
Unblock group of consumers on super stream partition (backport #7765)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A group of consumers on a super stream can end up blocked without an active consumer. This can happen with consumer churn: one consumer gets removed, which makes the active consumer passive, but the former active consumer never gets to know because it has been removed itself.
This commit changes the structure of the messages the SAC coordinator sends to consumer connections, to embed enough information to look up the group and to instruct it to choose a new active consumer when the race condition mentioned above comes up.
Because of the changes in the structure of messages, a feature flag is required to make sure the SAC coordinator starts sending the new messages only when all the nodes have been upgraded.
References #7743