Skip to content

Improve: Ensure uniform membership config during config changes #1351

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

drmingdrmer
Copy link
Member

@drmingdrmer drmingdrmer commented Apr 2, 2025

Changelog

Improve: Ensure uniform membership config during config changes

This commit improves the handling of membership configuration changes to
ensure that the system reliably transitions to a uniform configuration
in the second step of a joint membership change.

Problem:

During a membership config change, there are two steps:

  1. Transition to a joint configuration containing both Cold (current
    config) and Cnew (new config).
  2. Transition to a uniform configuration containing only Cnew.

Previously, the second step attempted to apply the same change on the
current membership state. If multiple membership change requests were
processed in parallel, this approach could result in the system being
left in a joint configuration. For example:

  • Initial config: {a, b, c}.

  • Task 1: AddVoterIds(x). After the first step:
    [{a, b, c}, {a, b, c, x}].

  • Task 2: RemoveVoters(x). After the first step:
    [{a, b, c, x}, {a, b, c}] (applied on the last config {a, b, c, x}).

  • Task 1 proceeds to the second step, re-applies AddVoterIds(x), and
    the result is [{a, b, c}, {a, b, c, x}].

  • The system remains in a joint configuration, which is unintuitive and
    contradicts standard Raft expectations.

Solution:

The second step now applies an empty change request
(AddVoterIds({})) to the last configuration in the current joint
config. This ensures that the system always transitions to a uniform
configuration in the second step.

Impact:

  • No behavior changes occur if only one membership change request is in
    progress.

  • If multiple requests are processed concurrently, the application must
    still verify the result, and the new behavior ensures the system
    transitions to a uniform state.

This fix prevents the system from being left in an unintended joint
configuration, improving consistency and adherence to Raft principles.

Thanks to @tvsfx for providing feedback on this issue and offering a
detailed explanation of the solution.


This change is Reviewable

@tvsfx
Copy link
Contributor

tvsfx commented Apr 2, 2025

That's a nice, minimal way to implement this flattening operation :) Now we just need to think of a way to make the retain parameter work better

@drmingdrmer drmingdrmer force-pushed the 195-change-membership-batch branch 2 times, most recently from 616301b to 27dbd28 Compare April 2, 2025 16:10
This commit improves the handling of membership configuration changes to
ensure that the system reliably transitions to a uniform configuration
in the second step of a joint membership change.

### Problem:

During a membership config change, there are two steps:

1. Transition to a joint configuration containing both `Cold` (current
   config) and `Cnew` (new config).
2. Transition to a uniform configuration containing only `Cnew`.

Previously, the second step attempted to apply the same change on the
current membership state. If multiple membership change requests were
processed in parallel, this approach could result in the system being
left in a joint configuration. For example:

- Initial config: `{a, b, c}`.
- Task 1: `AddVoterIds(x)`. After the first step:
  `[{a, b, c}, {a, b, c, x}]`.

- Task 2: `RemoveVoters(x)`. After the first step:
  `[{a, b, c, x}, {a, b, c}]` (applied on the last config `{a, b, c, x}`).

- Task 1 proceeds to the second step, re-applies `AddVoterIds(x)`, and
  the result is `[{a, b, c}, {a, b, c, x}]`.

- The system remains in a joint configuration, which is unintuitive and
  contradicts standard Raft expectations.

### Solution:

The second step now applies an **empty change request**
(`AddVoterIds({})`) to the last configuration in the current joint
config. This ensures that the system always transitions to a uniform
configuration in the second step.

### Impact:

- No behavior changes occur if only one membership change request is in
  progress.

- If multiple requests are processed concurrently, the application must
  still verify the result, and the new behavior ensures the system
  transitions to a uniform state.

This fix prevents the system from being left in an unintended joint
configuration, improving consistency and adherence to Raft principles.

Thanks to @tvsfx for providing feedback on this issue and offering a
detailed explanation of the solution.
@drmingdrmer drmingdrmer force-pushed the 195-change-membership-batch branch from 27dbd28 to 1e5aee2 Compare April 2, 2025 16:11
@tvsfx
Copy link
Contributor

tvsfx commented Apr 2, 2025

Oh, I just had a thought: I don't think this implementation works for removing Nodes and Voters as part of a batch at the same time: nodes can only be removed once the corresponding voter has been removed from the config already.

@drmingdrmer drmingdrmer marked this pull request as draft April 2, 2025 16:29
@drmingdrmer
Copy link
Member Author

Oh, I just had a thought: I don't think this implementation works for removing Nodes and Voters as part of a batch at the same time: nodes can only be removed once the corresponding voter has been removed from the config already.

Hmm... correct

@drmingdrmer
Copy link
Member Author

@tvsfx
It looks like this PR should be closed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants