Gain insights into distributed systems, learn architectural patterns like replication and sharding, and explore Apache Spark and Druid to design and develop effective distributed applications.

data_eng.tar.gz

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

Apache_Druid

Mapreduce

aws_load_balancer

service_sharding

cqrs

apache_spark

apache_kafka

Gone are the days when monolithic applications were the norm. Most applications that we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems in order to succeed.

In this course, you’ll learn what a distributed system is, followed by the challenges unique to distributed systems. You’ll cover popular architectural patterns for achieving targets like replication and sharding. You’ll finish with a study of two popular distributed systems: Apache Spark and Apache Druid.

With the knowledge gained through this course, you’ll be able to design and develop distributed systems. You’ll be able to make optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

As a case study of a distributed system, we will discuss the architecture of Apache Druid, a popular OLAP database.

# High-level architecture of Druid

Druid is an OLAP database. For systems that need support for a large volume of data, Druid has to be deployed as a distributed system. In a distributed environment, Druid has multiple nodes that work together to achieve the common goal of efficient analytical data querying.

In a distributed Druid system we have five types of nodes, as depicted in the following table:



align-center

The above five node types are internal to the Druid architecture. Apart from them, Druid also takes support from some external systems.

- **Deep storage**: To ensure fault tolerance, Druid uses deep storage. Any data in the system is stored in deep storage like AWS S3. This ensures data durability, as well as data transfer between different node types.

- **Metadata storage**: Holds metadata related to the data and the cluster. Druid stores important metadata so that data can be located in the cluster very quickly, which is a must for efficient queries.

- **ZooKeeper**: A third-party system that is used for cluster management.

The above is just a brief introduction to the various components of Apache Druid. We need to put them all together to understand how the system is organized.


Get to know the architecture of Apache Druid.

Type	Role
MiddleManagers	Ingest data into the Druid system
Historical nodes	Store the queryable data
Coordinators	Manage data availability in a Druid cluster by rebalancing and replication of data in Historical nodes
Overlords	Manage data ingestion by assigning ingestion tasks to MiddleManagers
Brokers	Handle queries from external systems or clients

Introduction

What Distributed Systems Achieve for Us

Data in Distributed Systems

Communication Between Nodes

Data Processing in Large Scale

Distributed System Architectural Patterns

Case Study 1: Apache Spark

Case Study 2: Apache Druid

Conclusion

Architecture of Apache Druid

High-level architecture of Druid