Gain insights into distributed systems, learn architectural patterns like replication and sharding, and explore Apache Spark and Druid to design and develop effective distributed applications.

data_eng.tar.gz

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

Apache_Druid

Mapreduce

aws_load_balancer

service_sharding

cqrs

apache_spark

apache_kafka

Gone are the days when monolithic applications were the norm. Most applications that we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems in order to succeed.

In this course, you’ll learn what a distributed system is, followed by the challenges unique to distributed systems. You’ll cover popular architectural patterns for achieving targets like replication and sharding. You’ll finish with a study of two popular distributed systems: Apache Spark and Apache Druid.

With the knowledge gained through this course, you’ll be able to design and develop distributed systems. You’ll be able to make optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

When we visit a webpage for the first time, it takes some time to load. But from that point on, it is quicker to load. 

❓ Do you ever wonder why this happens? The answer is **caching**.

In this lesson, we’ll discuss caching at the database level because this is a very well-known practice in the world of distributed systems. But remember that the concept is not limited to databases only. Your laptop itself has a cache that is an integral part of the machine’s hardware.

# What is caching?

The first time a website is accessed, your browser locally stores some data that helps to load the website faster in the future. This is the basic idea of caching.

> **Note**: In caching, frequently used data is stored in faster storage (like RAM) so that it can be quickly retrieved. The primary purpose of caching is to facilitate fast read latency.

Databases may use caching by default. When some query is executed frequently, and there is no change in data, the cache can help to execute the query even faster. This is very helpful in terms of performance and efficient resource usage.

We can take this even further. Sometimes, if the query pattern and use cases support it, we won’t even need to execute the query on the database. We can have a cache on the server-side itself. 

Caching can be as simple as storing some frequently accessed data in the memory of a server node in a hash table.

# Caching in a backend system

We know what a very simple system architecture looks like.

Learn about caching in the context of a distributed system.

Introduction

What Distributed Systems Achieve for Us

Data in Distributed Systems

Communication Between Nodes

Data Processing in Large Scale

Distributed System Architectural Patterns

Case Study 1: Apache Spark

Case Study 2: Apache Druid

Conclusion

Caching