Data replication: Master-Slave

Replication means keeping a copy of the same data on multiple machines that are connected via a network. Replication can be synchronous or asynchronous.

For very large datasets, or very high query throughput, that are not sufficient, we need to break the data up into partitions. This is also known as sharding.

Benefits

• Keeps data geographically close to your users, thus reducing access latency.

• Allows the system to continue working even if some of its parts have failed, thus increasing availability.

• Scales out the number of machines that can serve read queries, thus increasing read throughput.

Keep in mind

If the data that you are replicating does not change over time, then the replication process is a one-time thing. Frequently changing data is a real challenge; it requires careful thinking about concurrency and all the things that can go wrong so that we can deal with the consequences of those faults. At a minimum, we need to deal with unavailable nodes, network interruptions, and silent data corruption due to application bugs.

Here are the following algorithms for replicating changes across nodes:

• Master-Slave Replication (simple)

• Multi-leader replication (more robust)

• Leaderless replication

Master-Slave Replication

When a client wants to read from the database, it can query either the master or any of the slaves. However, writes are only accepted on the master. Often, leader-based replication is configured to be completely asynchronous. In this case, if the master fails and is not recoverable, any writes that have not yet been replicated to slaves are lost. This means that a write is not guaranteed to be durable. Weakening durability is a bad trade-off.

How can we come up with replication methods that do not lose data but still provide good performance and availability?

Handling slave node outages

Any node in the system can go down due to a fault or a reboot for maintenance purposes. On its local disk, each slave keeps a replication log of the data changes that it has received from the master. If the slave crashes or is restarted, it can recover from its local log as it knows the last transaction that was processed before the shutdown. Once the slave is up, it connects to the master and syncs up where it was left out.

Handling master node outages

The handling failure of the master node is trickier. Now, we need to upgrade the most updated slave as a new master and re-configure logic to write requests to a new master node.

In case of a semi-synchronous replication, we make the synchronous slave as a new master since we know that it is the most updated one and no data will be lost.

In case of asynchronous replication, there is some chance that the new master may not receive all the writes from the old master (assume this is still down). Hence, those write changes will be discarded, which could impact other listening applications and end clients. There are no easy solutions to this problem. For this reason, some operations teams prefer to perform fail-overs manually. So, is it better to use Multi-leader replication instead?

Understand replication log: Statement based replication

In the simplest case, the leader logs every write request (statement) that it executes and sends that statement log to its followers.This means that, for a relational database, every INSERT, UPDATE, or DELETE statement is forwarded to followers. Each follower then parses and executes that SQL statement as if it had been received from a client. Something to watch out for:

Any statement that calls a nondeterministic function, such as NOW(), to get the current date and time or RAND() to get a random number, is likely to generate a different value on each replica. The solution is for the leader to replace any non-deterministic function calls with a fixed return value when the statement is logged so that the followers all get the same value.

Best case usage

This type of replication requires all writes to go through a single master node, but read-only queries can go to any replicas. For workloads that consist of mostly reads and only a small percentage of writes (a common pattern on the web), there is an attractive option to create many followers and distribute the read requests across those followers. This removes a load from the leader and allows read requests to be served by nearby replicas.

Common problems using asynchronous master slave replication

• There is some chance that the new master may not receive all the writes from the old master (assume this is still down). Hence, those write changes will be discarded, which could impact other listening application and end clients.

• If the user tries to read the data immediately after making a write, the new data may not yet have reached the replica. However, to the user, it will look as though the data they submitted was lost.

Solution: When reading something that the user may have modified, read it from the leader; otherwise, read it from a follower. This requires that you have some way of knowing if something has been modified without actually querying it. For example, user profile information on a social network is normally only editable by the owner of the profile and not by anybody else. Thus, a simple rule is to always read the user’s own profile from the leader and to read any other users’ profiles from a follower.

Cons: This adds more stress on the master node

Multi-Leader replication

This replication process is more complicated than a master-slave, but it shines when we are dealing with multiple data centers because you can have a leader on each data center. Every leader sends its writes to every other leader (all-to-all topologyit allows messages to travel along different paths, thus avoiding a single point of failure)

Common data distribution/migration pattern

• Bi-direction: reporting instance

• Uni- direction: Instant fail-over ( Multi-leader replication)

• Peer to peer: Load balancing ( High Availability )

• Broadcast: wide level data distribution to multiple instances

• Consolidation: data warehouse (data storage )