Gain insights into distributed systems, learn architectural patterns like replication and sharding, and explore Apache Spark and Druid to design and develop effective distributed applications.

data_eng.tar.gz

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

Apache_Druid

Mapreduce

aws_load_balancer

service_sharding

cqrs

apache_spark

apache_kafka

Gone are the days when monolithic applications were the norm. Most applications that we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems in order to succeed.

In this course, you’ll learn what a distributed system is, followed by the challenges unique to distributed systems. You’ll cover popular architectural patterns for achieving targets like replication and sharding. You’ll finish with a study of two popular distributed systems: Apache Spark and Apache Druid.

With the knowledge gained through this course, you’ll be able to design and develop distributed systems. You’ll be able to make optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

# What and why of replication

> Replication means keeping a copy of the same data on multiple machines that are
connected via a network.
> <div align="right""> <em>Designing Data-Intensive Application</em>, M Kleppmann </div>


There are two parts to this definition.

1. **Keeping the same data on multiple machines**
   - It’s required to store the copies that are exactly the same as the original data everywhere. The purpose is to use multiple machines to serve users. We would want to handle the increasing volume of reads and writes using more machines, as discussed previously.

2. **The machines are connected via a  network**

   - If the machines are not connected via a network, then they cannot communicate creations, changes, or updates to the data. Remember that the machines need to have the exact same data stored in them. To keep them always in sync, there has to be network communication.

   - Of course, keeping the data the same across machines is easy if your data is static and never changes. You can just run multiple machines with a copy of the same data and serve the clients. Network communication between the storage nodes is not required.

In distributed systems, we need data replication to still work even if it partially fails. Also, with replication, we can scale the system when the business grows. Most systems do have data that changes over time. For many systems, network communication between storage nodes is a must.

Now, this might sound so simple—to have a couple of nodes and copy data from one to another via a network—but unfortunately replication is a difficult problem to solve in a system that requires it. Networks are unreliable. Machines fail randomly, and storage capacity has its limits. With more data comes more problems.

# How to do replication?

Imagine that the data of the My Cool App has the following properties:

- The data has a large volume, possibly a few hundred gigabytes.
- We have such machines at our disposal that a single machine can hold the entirety of the data.
- Since a single machine is a single point of failure, and it hurts availability, we definitely require multiple machines and store copies of the same data in all of them.
- The data changes over time. New data gets added and updated all the time.

Now, given the above scenarios, we need to make sure that when anything changes in our data, all the machines containing a replica of the data get updated accordingly. If a new row gets added, or a column is updated, or some rows are deleted, we see all these changes reflected in every node. If one or a few nodes do not contain these changes in them, then requests served from these nodes will have outdated information, which basically hurts the reliability of the system altogether. As an owner, you will definitely not want this in your system.


> **Note**: Replication ensures data is correctly available in every part of the system so that requests distributed in different replica nodes have a consistent response. This directly increases the system's scalability.

# Key takeaways

- Replication means having multiple copies of the exact same data in multiple nodes so that failure in one or more does not necessarily impact the system operation.
- Like other aspects of a distributed system, with a larger volume of data, replication also becomes more complicated to handle.

Learn about replication in distributed systems.

Introduction

What Distributed Systems Achieve for Us

Data in Distributed Systems

Communication Between Nodes

Data Processing in Large Scale

Distributed System Architectural Patterns

Case Study 1: Apache Spark

Case Study 2: Apache Druid

Conclusion

Replication

What and why of replication