Gain insights into distributed systems, learn architectural patterns like replication and sharding, and explore Apache Spark and Druid to design and develop effective distributed applications.

data_eng.tar.gz

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

Apache_Druid

Mapreduce

aws_load_balancer

service_sharding

cqrs

apache_spark

apache_kafka

Gone are the days when monolithic applications were the norm. Most applications that we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems in order to succeed.

In this course, you’ll learn what a distributed system is, followed by the challenges unique to distributed systems. You’ll cover popular architectural patterns for achieving targets like replication and sharding. You’ll finish with a study of two popular distributed systems: Apache Spark and Apache Druid.

With the knowledge gained through this course, you’ll be able to design and develop distributed systems. You’ll be able to make optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

There are three approaches to conducting replication in a distributed system:

- Single-leader replication 
- Multi-leader replication
- Leaderless replication

Let's explore these replication techniques.

# Single-leader replication

Each node storing a copy of the database is called a **replica**. A database system may have many replicas. With replication, these replica nodes are in sync, and they remain consistent over time.

In single-leader replication, one of the replicas is called the **leader**. All writes from clients are handled by the leader.

Other replicas are called **followers**. Followers do not handle writes at all. They keep in sync with the leader and serve read requests to the clients.

Briefly, this is how single-leader replication works:


- Upon receiving a write-request, the leader stores the data in its local storage.
- The leader also communicates the write-request to the followers.
- The followers process the write on the local copy they have.
- When a client wants to read some data, it can connect to any of the replicas, including the leader.

Many databases, such as MySQL, use a single-leader replication mechanism. This means all writes in MySQL will be done via the leader node, and reads can be served by any node. This is why in many cases MySQL is suitable for read-heavy systems. This is also the reason why MySQL is difficult to scale for a large volume of data.

The drawback of single-leader replication is that if clients are unable to connect to the leader, then the writes cannot proceed.

# Multi-leader replication

In multi-leader replication, there is more than one leader in the system. The mechanism is pretty much the same as single-leader replication:

- There are multiple leaders in the system and each leader has its own set of followers exclusive to it.
- A write-request reaches any one of the leaders.
- Upon receiving the write-request, that particular leader writes data in its own local storage.
- The leader then communicates the write to its own followers, including other leaders.
- Other leaders process the write on their side and let the write proceed to their followers.
- Any read can be done on any of the replicas.


Gain an understanding of different replication techniques in distributed systems.

Introduction

What Distributed Systems Achieve for Us

Data in Distributed Systems

Communication Between Nodes

Data Processing in Large Scale

Distributed System Architectural Patterns

Case Study 1: Apache Spark

Case Study 2: Apache Druid

Conclusion

Replication Techniques

Single-leader replication