Gain insights into Redis's capabilities as a database, cache, and message broker. Delve into commands, transactions, and clustering. Explore flexibility, security, and data structure support with Redis.

Redis.tar.gz

Redis is an open-source, in-memory data structure store used as a database, cache, and message broker. Unlike Memcached, it supports multiple data types, giving a lot of flexibility to the user. Redis is easy to configure and provides clustering support to store data in multiple servers.

In this course, you will learn about the various commands that you can use to store different types of data structures in Redis. Some of those commands include utility, insertion, and modification commands.

You will also become familiar with different aspects of Redis, such as transactions, security, partitioning, and clustering. In all, this is your go-to guide to Redis.

Complete Guide to Redis

## What is data replication?
When data is stored on a server and there is a server crash, there is a chance that the data will be lost. To avoid this, the technique of data replication is used. The data is stored on two or more servers instead of a single server. That way, if a server goes down, data is not lost since it is available on the other server. Also, the application does not crash because the other server can cater to the user requests. Another benefit of data replication is that it can reduce the load on the servers because the user requests can be load balanced to the servers instead of sending it to a single server.



Redis follows a *master-slave* approach for replication. One of the servers is a master, and the other servers are called slaves. The slaves are connected to the master. All the writes happen to the master and then the master, sends these changes to the slaves. The reads can then be handled by the slaves. Through this process, the load is distributed.

If a slave gets disconnected from the master, it automatically reconnects and tries to be the exact copy of the master. There are two approaches that a slave takes to get in sync with the master.
* **Partial Synchronization**: The slave tries to get only the data that it missed while it was disconnected. The slave sends the master information about the point to which it has the data. We will discuss this in more detail later.
* **Full Synchronization**: If partial synchronization is not possible, then full synchronization takes place. The master sends all the data to an rdb file. This rdb file is sent to the slave. The slave then saves it to its disk. After that is done, the slave loads the data from the rdb file to its memory. There is an option of *disk-less replication* as well. The child process can directly send the rdb to replicas over the wire, without using the disk as intermediate storage.

## Synchronization of master and salves

1. The master is non-blocking and continues to handle queries when one or more replicas perform the initial synchronization or a partial re-synchronization.
2. The slave is also non-blocking. When a slave is synchronizing the data, it will also serve the clients from the old data if it is configured to do so.
3. When a slave gets disconnected from the master, it may or may not serve the clients. It depends on the `slave-serve-stale-data` configuration. If this configuration is set to yes, then the slave will serve out-of-date data to the client. If it is false, then the slave will reply with a **SYNC with master in progress**  error to all the commands except INFO and SLAVEOF.
4. The slave sends a ping to the master after regular intervals. This tells the master if a slave is still connected or not. By default, a slave sends the ping every ten seconds, but this can be configured using the `repl-ping-slave-period` property in the redis.conf file.
5. We can configure how much bandwidth is used in syncing the slave by using the `repl-disable-tcp-nodelay` configuration. If this value is **yes**, Redis will use a smaller number of TCP packets and less bandwidth to send data to slaves. This will delay the appearance of data on the slave side. If **no** is selected, then the delay will be reduced, but more bandwidth will be used for replication.
6. When a slave disconnects from a master, the data being transmitted is saved in a buffer. When a slave reconnects, it gets the data from this buffer. If a slave is disconnected for too long, this buffer may not have all the data that the slave missed, so full sync is required. The larger the size of the buffer, the higher the chances are of a partial sync. The default size of the buffer is 1 MB, but this can be configured using `repl-backlog-size`. The buffer is allocated only when the first slave is connected to the master.
7. If all the slaves are disconnected from the master, then the buffer is not required anymore. It is deleted after a certain period of time, which can be defined through the `repl-backlog-ttl` property. The default value is 3,600 seconds.
8. Each slave has a priority that is used to decide which slave will be promoted to master in case the master crashes. If there are three slaves with priorities 20, 30, and 50 then the slave with priority 20 will be promoted to master. The default priority of a server is 100. This can be configured using the `slave-priority` configuration. If this field is set to 0, then it means that a slave cannot be promoted to master.

## Performance impact of replication in Redis
Since Redis stores data in-memory, it needs to store the data on the disk from time to time in order to protect the data in case of a failure. Storing the data on the disk is a costly operation so it is possible that, due to latency concerns, persistence is disabled on the master server. However, turning the persistence off is not advised unless it is absolutely necessary.  If disabling persistence is necessary, the master should be configured to avoid restarting automatically after a reboot. 

If persistence is disabled and the server restarts automatically, it may lead to data loss. If this happens, the slaves will try to sync with the master, and their data will also be lost.

## How replication is done in Redis?
The replication process in Redis is asynchronous. The master sends the data to be replicated to the slave and does not check if it was actually saved in the slave server. The slave server asynchronously acknowledges the amount of data received periodically from the master. This gives the master an idea of what commands the slaves have processed.

We discussed earlier that if a slave is disconnected and reconnected, it syncs only the data that it missed. The question that arises here how the master knows what data the slave missed. The **replication ID** and **offset** are used to figure this out. Each master server has a **replication ID** that is also inherited by a slave. The master also has an offset that increments for every byte of replication stream that is produced to be sent to replicas. When a slave is reconnected, it sends its offset to the server.

Let's suppose the master has a replication ID of **123gh3** and an offset as **12,000**. The slave has the replication ID of **123gh3** and the offset of **10,000**. When the slave connects with the master, it will send its offset of 10,000 to the master. The master will then know which bytes the slave is missing and it will resend those bytes to the slave.

There can be a situation where the offset provided by the slave is too old, and the master does not have that stream in the buffer. In that case, partial sync cannot be done and full sync is required.





# What is data replication?
When data is stored on a server and there is a server crash, there is a chance that the data will be lost. To avoid this, the technique of data replication is used. The data is stored on two or more servers instead of a single server. That way, if a server goes down, data is not lost since it is available on the other server. Also, the application does not crash because the other server can cater to the user requests. Another benefit of data replication is that it can reduce the load on the servers because the user requests can be load balanced to the servers instead of sending it to a single server.



Redis follows a *master-slave* approach for replication. One of the servers is a master, and the other servers are called slaves. The slaves are connected to the master. All the writes happen to the master and then the master, sends these changes to the slaves. The reads can then be handled by the slaves. Through this process, the load is distributed.

If a slave gets disconnected from the master, it automatically reconnects and tries to be the exact copy of the master. There are two approaches that a slave takes to get in sync with the master.
* **Partial Synchronization**: The slave tries to get only the data that it missed while it was disconnected. The slave sends the master information about the point to which it has the data. We will discuss this in more detail later.
* **Full Synchronization**: If partial synchronization is not possible, then full synchronization takes place. The master sends all the data to an rdb file. This rdb file is sent to the slave. The slave then saves it to its disk. After that is done, the slave loads the data from the rdb file to its memory. There is an option of *disk-less replication* as well. The child process can directly send the rdb to replicas over the wire, without using the disk as intermediate storage.

# Synchronization of master and salves

1. The master is non-blocking and continues to handle queries when one or more replicas perform the initial synchronization or a partial re-synchronization.
2. The slave is also non-blocking. When a slave is synchronizing the data, it will also serve the clients from the old data if it is configured to do so.
3. When a slave gets disconnected from the master, it may or may not serve the clients. It depends on the `slave-serve-stale-data` configuration. If this configuration is set to yes, then the slave will serve out-of-date data to the client. If it is false, then the slave will reply with a **SYNC with master in progress**  error to all the commands except INFO and SLAVEOF.
4. The slave sends a ping to the master after regular intervals. This tells the master if a slave is still connected or not. By default, a slave sends the ping every ten seconds, but this can be configured using the `repl-ping-slave-period` property in the redis.conf file.
5. We can configure how much bandwidth is used in syncing the slave by using the `repl-disable-tcp-nodelay` configuration. If this value is **yes**, Redis will use a smaller number of TCP packets and less bandwidth to send data to slaves. This will delay the appearance of data on the slave side. If **no** is selected, then the delay will be reduced, but more bandwidth will be used for replication.
6. When a slave disconnects from a master, the data being transmitted is saved in a buffer. When a slave reconnects, it gets the data from this buffer. If a slave is disconnected for too long, this buffer may not have all the data that the slave missed, so full sync is required. The larger the size of the buffer, the higher the chances are of a partial sync. The default size of the buffer is 1 MB, but this can be configured using `repl-backlog-size`. The buffer is allocated only when the first slave is connected to the master.
7. If all the slaves are disconnected from the master, then the buffer is not required anymore. It is deleted after a certain period of time, which can be defined through the `repl-backlog-ttl` property. The default value is 3,600 seconds.
8. Each slave has a priority that is used to decide which slave will be promoted to master in case the master crashes. If there are three slaves with priorities 20, 30, and 50 then the slave with priority 20 will be promoted to master. The default priority of a server is 100. This can be configured using the `slave-priority` configuration. If this field is set to 0, then it means that a slave cannot be promoted to master.

# Performance impact of replication in Redis
Since Redis stores data in-memory, it needs to store the data on the disk from time to time in order to protect the data in case of a failure. Storing the data on the disk is a costly operation so it is possible that, due to latency concerns, persistence is disabled on the master server. However, turning the persistence off is not advised unless it is absolutely necessary.  If disabling persistence is necessary, the master should be configured to avoid restarting automatically after a reboot. 

If persistence is disabled and the server restarts automatically, it may lead to data loss. If this happens, the slaves will try to sync with the master, and their data will also be lost.

# How replication is done in Redis?
The replication process in Redis is asynchronous. The master sends the data to be replicated to the slave and does not check if it was actually saved in the slave server. The slave server asynchronously acknowledges the amount of data received periodically from the master. This gives the master an idea of what commands the slaves have processed.

We discussed earlier that if a slave is disconnected and reconnected, it syncs only the data that it missed. The question that arises here how the master knows what data the slave missed. The **replication ID** and **offset** are used to figure this out. Each master server has a **replication ID** that is also inherited by a slave. The master also has an offset that increments for every byte of replication stream that is produced to be sent to replicas. When a slave is reconnected, it sends its offset to the server.

Let's suppose the master has a replication ID of **123gh3** and an offset as **12,000**. The slave has the replication ID of **123gh3** and the offset of **10,000**. When the slave connects with the master, it will send its offset of 10,000 to the master. The master will then know which bytes the slave is missing and it will resend those bytes to the slave.

There can be a situation where the offset provided by the slave is too old, and the master does not have that stream in the buffer. In that case, partial sync cannot be done and full sync is required.





In this lesson, we will discuss how data is replicated in Redis.

Let's Begin

Redis Commands

Redis Advanced Concepts

Redis Cluster

Appendix

Redis: Replication

What is data replication?

Replication in Redis