Enable Fault Tolerance and Failure Detection

Handle temporary failures

Typically, distributed systems use a quorum-based approach to handle failures. A quorum is the minimum number of votes required for a distributed transaction to proceed with an operation. If a server is part of the consensus and is down, then we can’t perform the required operation. It affects the availability and durability of our system.

We’ll use a sloppy quorum instead of strict quorum membership. Usually, a leader manages the communication among the participants of the consensus. The participants send an acknowledgment after committing a successful write. Upon receiving these acknowledgments, the leader responds to the client. However, the drawback is that the participants are easily affected by the network outage. If the leader is temporarily down and the participants can’t reach it, they declare the leader dead. Now, a new leader has to be reelected. Such frequent elections have a negative impact on performance because the system spends more time picking a leader than accomplishing any actual work.

In the sloppy quorum, the first $n$ healthy nodes from the preference list handle all read and write operations. The $n$ healthy nodes may not always be the first $n$ nodes discovered when moving clockwise in the consistent hash ring.

Let’s consider the following configuration with $n = 3$. If node $A$ is briefly unavailable or unreachable during a write operation, the request is sent to the next healthy node from the preference list, which is node $D$ in this case. It ensures the desired availability and durability. After processing the request, the node $D$ includes a hint as to which node was the intended receiver (in this case, $A$). Once node $A$ is up and running again, node $D$ sends the request information to $A$ so it can update its data. Upon completion of the transfer, $D$ removes this item from its local storage without affecting the total number of replicas in the system.