Raft's Cluster Membership Changes

Cluster membership changes

Previously, it was assumed that the configuration of servers participating in the consensus algorithm is fixed. Sometimes, it is necessary to change the configuration to replace failed servers or change the replication levels of servers. However, taking the entire cluster offline to update configuration files is risky. The associated risk could be an operator error causing unavailability. To automate configuration changes and avoid these issues, the Raft consensus algorithm incorporates a mechanism for configuration changes.

Raft’s configuration change process must provide safety, preventing the possibility of two servers getting elected as leaders for the same term. However, directly converting all servers from the old configuration to the new one is risky since it risks dividing the cluster into two isolated majorities. As a result, switching all servers atomically at once is not possible.

The following illustration depicts an issue: If we switch directly from one configuration to another, different servers will make that switch randomly. There must be a moment when two majorities would follow two different cluster configurations, possibly having two different leaders from two different configurations.

Joint consensus configuration

Raft uses a two-phase approach to ensure safety during configuration changes:

• Phase I: It first switches the cluster to a transitional configuration called joint consensus, which combines the old and new configurations with the following rules.
  • Log entries are replicated to all servers in both configurations.
  • Any server from either configuration may serve as the leader.
  • Agreement for elections and entry commitment requires separate majorities from old and new configurations.
• Phase II: It then transitions into the new configuration by copying the new configuration file to all the servers.

The cluster’s configurations are stored and communicated through special log entries, separate from regular ones, as shown in the illustration below:

1. An admin client can request the current leader to initiate configuration change. When the leader requests to change the configuration from $C_{old}$ to $C_{new}$, it stores the new configuration as a log entry for joint consensus $C_{old,\ new}$