Gain insights into distributed systems, learn architectural patterns like replication and sharding, and explore Apache Spark and Druid to design and develop effective distributed applications.

data_eng.tar.gz

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

Apache_Druid

Mapreduce

aws_load_balancer

service_sharding

cqrs

apache_spark

apache_kafka

Gone are the days when monolithic applications were the norm. Most applications that we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems in order to succeed.

In this course, you’ll learn what a distributed system is, followed by the challenges unique to distributed systems. You’ll cover popular architectural patterns for achieving targets like replication and sharding. You’ll finish with a study of two popular distributed systems: Apache Spark and Apache Druid.

With the knowledge gained through this course, you’ll be able to design and develop distributed systems. You’ll be able to make optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

One very popular and interesting consistency is **tunable consistency**. The name suggests the idea: we can tune our consistency to the type we would like to achieve in our system.

Databases such as Apache Cassandra have largely popularized this consistency model. Over the years, many NoSQL databases have adopted this strategy.

Let’s discuss tunable consistency and its associated concepts.



# Consistency level

Consistency level describes the number of nodes that must acknowledge a read or write before a request is marked as successful. We need to consider two types of consistency levels:

- **Read consistency level** denotes the number of replica nodes that must respond to the read request with the latest data they have.
- **Write consistency level** denotes how many replica nodes should acknowledge that the request has been processed on their side.

# Coordinator node

In a distributed database cluster, the coordinator node is the node that receives clients’ requests and dispatches them to the correct nodes. The client can send its requests to any node in the cluster. It is the responsibility of the cluster to handle the request correctly.

We discussed the same concept in [leaderless replication](https://www.educative.io/courses/introduction-to-distributed-systems-for-dummies/replication-techniques).

# Parameters

Let’s define some parameters to help your understanding of tunable consistency.

- $R$: The number of minimum replica nodes that the coordinator will read from for serving a read request. This is basically the read consistency level.
- $W$: The number of minimum replica nodes that must acknowledge a write-request to the coordinator. This is the write consistency level.
- $RF$: The replication factor. This is the number of nodes the partitions will be replicated to.
- $QUORUM$: This denotes the number of majority nodes derived from the replication factor. If a cluster has  $RF$ equal to $5$ nodes, then $QUORUM = 3$. For $RF = 3$, $QUORUM = 2$.

> $QUORUM = (\frac{RF}{2}) + 1$ 

# Tuning consistency

Now that we understand the terms required, let’s discuss how to tune the consistency model.

## Strong consistency

> $R + W > RF$

Assume that $RF$ is $3$. So, data is replicated into $3$ nodes. Now we need the summation of $R$ and $W$ to be greater than $3$.

As an example, if $W = QUORUM$ and $R = QUORUM$, then we have $R + W = 4 > 3$. This ensures that at least one node is going to serve the latest data,  ensuring strong consistency. How is that so?

> Assume you have three nodes in your system -- $n_0$, $n_1$, and $n_2$. Also, $RF$ is $3$. So,
>
> $QUORUM = 2$ \
> $R = 2$ \
> $W = 2$
>
> Say, the write for a key is acknowledged by $n_0$ and $n_1$, and read is responded from $n_1$ and $n_2$. So, there is node $n_1$, that will have the latest updates.

Now we can set $R$ and $W$ in a way that either read is optimized, or write is optimized, or both are balanced. For instance:

- Setting both the values as $QUORUM$ gives a balanced performance. 
- Setting $R = 1$ and $W = RF$ makes read-requests fast but write-requests slow.
- Similarly, $R = RF$ and $W = 1$ gives us a fast write performance but a slow read performance.

How you set the parameters strongly depends on what your system type and business requirements are.

## Eventual consistency

Not all systems require strong consistency. We can always tune the parameters to go for eventual consistency in our system.

> $R + W \leq RF$

Similarly, as above, assume $RF = 3$. And we set $R = QUORUM$ and $W = 1$. This results in a chance of getting data from nodes that do not contain the latest changes, resulting in eventual consistency.

>Just like above, our cluster has $3$ nodes -- $n_0$, $n_1$ and $n_2$. $RF$ is set to $3$. Now, 
>
> $QUORUM = 2$ \
> $R = 2$ \
> $W = 1$
> 
> Say, write is acknowledged by $n_2$, but the read is responded by $n_0$ and $n_1$. So, we get outdated updates.

Given the formula above, we can set $R$ and $W$ loosely. For example, $R = 1, W = 1$. This gives us no guarantee that we can always get the latest update after a write. But since $RF = 3$, eventually, all nodes will be in sync.

Similar to strong consistency, the consistency level for reads and writes will determine how much latency we see for the requests.

# Key takeaways

- To achieve strong consistency, we need to beat the replication factor by ensuring that we write to enough nodes or read from enough nodes.
- In eventual consistency, we prefer speed which means data can be stale. This helps to achieve high availability.

Learn about tunable consistency in distributed systems.

Introduction

What Distributed Systems Achieve for Us

Data in Distributed Systems

Communication Between Nodes

Data Processing in Large Scale

Distributed System Architectural Patterns

Case Study 1: Apache Spark

Case Study 2: Apache Druid

Conclusion

Tunable Consistency

Consistency level

Coordinator node

Parameters