Gain insights into distributed systems, learn architectural patterns like replication and sharding, and explore Apache Spark and Druid to design and develop effective distributed applications.

data_eng.tar.gz

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

Apache_Druid

Mapreduce

aws_load_balancer

service_sharding

cqrs

apache_spark

apache_kafka

Gone are the days when monolithic applications were the norm. Most applications that we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems in order to succeed.

In this course, you’ll learn what a distributed system is, followed by the challenges unique to distributed systems. You’ll cover popular architectural patterns for achieving targets like replication and sharding. You’ll finish with a study of two popular distributed systems: Apache Spark and Apache Druid.

With the knowledge gained through this course, you’ll be able to design and develop distributed systems. You’ll be able to make optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

When we were discussing replication, we learned it takes time for the data to propagate from the leader to the followers. Data travels over the network, and each follower node has to process the write-request on their side. So intuitively, we can say that there is a lag.

This lag, known as the **replication lag**, is responsible for consistency issues in distributed systems. Based on the needs of your business, you will have to choose from different consistency models. 

Note that there are two levels of data consistency: the disk caching level and the database level. In the context of distributed systems, we will discuss consistency on the database level. Let’s begin.

# Consistency models

> **Consistency** is the property that denotes how “in-sync” data is among the storage nodes in a distributed system.

The definition might sound vague. The details will become clearer as we discuss the two types of consistency models.

## Strong consistency

The first consistency model we will discuss is **strong consistency**. 

If a system guarantees strong consistency, then all the nodes will always have the exact same copy of the data when a read request is fired at any point in time. So if there are $5$ nodes in the cluster, it does not matter which node is hit for fetching data from the system. All nodes are capable of serving the latest version of the data.

> **Note**: In strong consistency, the system appears to be holding one copy of data from the users’ perspective, even though there are multiple copies of the data across many nodes.

As we mentioned, there is a replication lag in followers when they process a write-request on their side. Now while they are processing the write-request, if a read-request comes for the same data to the follower nodes, inconsistent or stale data might be returned.

But there are some systems that cannot tolerate stale or inconsistent data, for example, financial systems that literally deal with users' money. Nobody would like to see inconsistent transactions or statements in such systems. This is why systems that guarantee strong consistency handle reads via a strict method.


Learn about consistency in distributed systems.

Introduction

What Distributed Systems Achieve for Us

Data in Distributed Systems

Communication Between Nodes

Data Processing in Large Scale

Distributed System Architectural Patterns

Case Study 1: Apache Spark

Case Study 2: Apache Druid

Conclusion

Consistency