Improve: Ensure uniform membership config during config changes

This commit improves the handling of membership configuration changes to
ensure that the system reliably transitions to a uniform configuration
in the second step of a joint membership change.

### Problem:

During a membership config change, there are two steps:

1. Transition to a joint configuration containing both `Cold` (current
   config) and `Cnew` (new config).
2. Transition to a uniform configuration containing only `Cnew`.

Previously, the second step attempted to apply the same change on the
current membership state. If multiple membership change requests were
processed in parallel, this approach could result in the system being
left in a joint configuration. For example:

- Initial config: `{a, b, c}`.
- Task 1: `AddVoterIds(x)`. After the first step:
  `[{a, b, c}, {a, b, c, x}]`.

- Task 2: `RemoveVoters(x)`. After the first step:
  `[{a, b, c, x}, {a, b, c}]` (applied on the last config `{a, b, c, x}`).

- Task 1 proceeds to the second step, re-applies `AddVoterIds(x)`, and
  the result is `[{a, b, c}, {a, b, c, x}]`.

- The system remains in a joint configuration, which is unintuitive and
  contradicts standard Raft expectations.

### Solution:

The second step now applies an **empty change request**
(`AddVoterIds({})`) to the last configuration in the current joint
config. This ensures that the system always transitions to a uniform
configuration in the second step.

### Impact:

- No behavior changes occur if only one membership change request is in
  progress.

- If multiple requests are processed concurrently, the application must
  still verify the result, and the new behavior ensures the system
  transitions to a uniform state.

This fix prevents the system from being left in an unintended joint
configuration, improving consistency and adherence to Raft principles.

Thanks to @tvsfx for providing feedback on this issue and offering a
detailed explanation of the solution.

Author: drmingdrmer

openraft/src/docs/cluster_control/joint-consensus.md

@@ -0,0 +1,43 @@
+# Joint Consensus in OpenRaft 
+
+## Introduction
+
+Joint consensus is a mechanism in the Raft protocol that allows for safe changes to cluster membership. When membership changes occur (such as adding or removing nodes), the cluster transitions through a joint configuration phase where both old and new configurations are considered valid. This ensures that the cluster maintains availability and safety during transitions.
+
+## Membership Change Process
+
+OpenRaft implements membership changes using a two-phase joint consensus approach:
+
+1. **First Phase:** Apply a membership change request (e.g., `AddVoters({x})`) to the current configuration, forming a joint configuration that contains both the old and new configurations. This joint configuration is then committed.
+
+2. **Second Phase:** Apply an **empty change request** (`AddVoterIds({})`) to the last configuration in the current joint config. This transitions the cluster from joint configuration to a uniform configuration and is then committed.
+
+## Problem in Previous Versions of OpenRaft (prior to 2025-04-03)
+
+In earlier implementations, the second phase of a membership change would reapply the original change operation to the current membership state. When multiple membership change requests were processed concurrently, this approach could leave the system in a joint configuration state rather than transitioning to a uniform configuration. Consider this example:
+
+- Initial configuration: `{a, b, c}`
+- Task 1: `AddVoterIds(x)` 
+  - After first phase: `[{a, b, c}, {a, b, c, x}]`
+- Task 2: `RemoveVoters(x)` (runs concurrently)
+  - After first phase: `[{a, b, c, x}, {a, b, c}]` (applied to the last configuration `{a, b, c, x}`)
+- Task 1 proceeds to second phase, reapplies `AddVoterIds(x)` to the current state
+  - Result: `[{a, b, c}, {a, b, c, x}]` (still a joint configuration)
+
+This behavior was problematic because:
+1. The system remained in a joint configuration state indefinitely
+2. This contradicted the standard Raft expectation that membership changes should eventually result in a uniform configuration
+3. It created confusion for users who expected membership changes to be fully applied
+
+## Solution
+
+The second step now applies an **empty change request** (`AddVoterIds({})`) to the last configuration in the current joint config. This ensures that the system always transitions to a uniform configuration in the second step, regardless of concurrent membership operations.
+
+## Impact
+
+- **Single Change Request:** No behavior changes occur if only one membership change request is in progress.
+- **Concurrent Requests:** If multiple requests are processed concurrently, the application must still verify the result, but the new behavior ensures the system always transitions to a uniform state.
+
+## Acknowledgments
+
+Thanks to @tvsfx for providing feedback on this issue and offering a detailed explanation of the solution.

openraft/src/docs/cluster_control/mod.rs

@@ -8,6 +8,10 @@ pub mod dynamic_membership {
     #![doc = include_str!("dynamic-membership.md")]
 }
 
+pub mod joint_consensus {
+    #![doc = include_str!("joint-consensus.md")]
+}
+
 pub mod node_lifecycle {
     #![doc = include_str!("node-lifecycle.md")]
 }

openraft/src/docs/docs.md

@@ -8,6 +8,7 @@ To maintain an Openraft cluster, e.g., add or remove nodes, refer to
 - [`cluster_control`](crate::docs::cluster_control) :
   - [`cluster_formation`](`crate::docs::cluster_control::cluster_formation`) describes how to form a cluster;
   - [`dynamic membership`](`crate::docs::cluster_control::dynamic_membership`) describes how to add or remove nodes without downtime;
+  - [`joint_consensus`](`crate::docs::cluster_control::joint_consensus`) describes detail of joint consensus implementation;
   - [`node lifecycle`](`crate::docs::cluster_control::node_lifecycle`) describes the transition of a node's state;
 
 When upgrading an Openraft application, consult:

openraft/src/raft/impl_raft_blocking_write.rs

@@ -31,6 +31,9 @@ where C: RaftTypeConfig<Responder = OneshotResponder<C>>
     /// - It proposes a **joint** config.
     /// - When the **joint** config is committed, it proposes a uniform config.
     ///
+    /// Read more about the behavior of [joint
+    /// consensus](crate::docs::cluster_control::joint_consensus).
+    ///
     /// If `retain` is `true`, then all the members which not exists in the new membership,
     /// will be turned into learners, otherwise will be removed.
     /// If `retain` is `false`, the removed voter will be removed from the cluster.
@@ -96,7 +99,19 @@ where C: RaftTypeConfig<Responder = OneshotResponder<C>>
 
         let (tx, rx) = oneshot_channel::<C>();
 
-        let res = self.inner.call_core(RaftMsg::ChangeMembership { changes, retain, tx }, rx).await;
+        // The second step, send a NOOP change to flatten the joint config.
+
+        let res = self
+            .inner
+            .call_core(
+                RaftMsg::ChangeMembership {
+                    changes: ChangeMembers::AddVoterIds(Default::default()),
+                    retain,
+                    tx,
+                },
+                rx,
+            )
+            .await;
 
         if let Err(e) = &res {
             tracing::error!("the second step error: {}", e);