Gain insights into distributed systems, learn architectural patterns like replication and sharding, and explore Apache Spark and Druid to design and develop effective distributed applications.

data_eng.tar.gz

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

Apache_Druid

Mapreduce

aws_load_balancer

service_sharding

cqrs

apache_spark

apache_kafka

Gone are the days when monolithic applications were the norm. Most applications that we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems in order to succeed.

In this course, you’ll learn what a distributed system is, followed by the challenges unique to distributed systems. You’ll cover popular architectural patterns for achieving targets like replication and sharding. You’ll finish with a study of two popular distributed systems: Apache Spark and Apache Druid.

With the knowledge gained through this course, you’ll be able to design and develop distributed systems. You’ll be able to make optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

When we discussed replication, we assumed that the My Cool App had such an amount of data that it was always possible to store the whole copy of the app in a single machine. But in reality, as a system grows, it is no longer feasible for all the data to be stored in a single machine. Businesses grow with time, new use-cases show up, data become an asset, and obviously, data size increases.

Apart from data size, sometimes supporting a large number of queries becomes pretty complicated if all the data is stored in one machine.

This is where **partitioning** is used.

# What is partitioning?
Partitioning is a mechanism in which data is divided into smaller chunks based on some specific attributes. These chunks are called partitions.

One partition is independent of another partition. Two partitions can be stored in two different machines in a distributed system. A partition can also be treated as a standalone database table.

Note that another well-known term for this concept is called **sharding**. Some people use these two terms interchangeably. But in general:

- Sharding denotes physically storing data in multiple server instances. 
- Partitioning refers to both the physical and the logical division of data. For example, if you break down a database table into multiple smaller tables but still store the tables on the same machine, this is partitioning.

Due to the genericness in the term _partitioning_, we will stick to it. In general, the difference between these two terms, from the perspective of different database technologies, is vague.

## An example of partitioning

Assume that in the My Cool App, users can upload images and add captions to images. Since the app is growing fast, one machine can no longer hold all the images your users are uploading. You need to spread the images on to different machines. This is where you can use partitioning.

Assuming your user IDs are integers, one possible strategy is to store a range of the first $n$ number of users' images in one machine, then the next $n$ users’ images in another one, and so on. This is just one simple strategy, and it may or may not be optimal.

So in the above example the data is divided into chunks so that the chunks are independent of one another. When you want to query for images for a user, you will have to route the requests to the proper machine so that data can be easily retrieved.

In MySQL, there is a `PARTITION BY` keyword which can be used to partition a table into multiple smaller tables based on some attribute. But the SQL query you will write is capable of treating all the partitions as a single table and executing the queries correctly. So basically, the SQL layer handles the underlying partitioned tables.

# Why choose partitioning in the first place?

Let's discuss briefly why we need partitioning:

- As mentioned already, in modern systems, data is unbounded. When the system grows, it becomes increasingly important to implement partitioning for the data. Otherwise, storing is bottlenecked by the machine’s constraints.
- Partitioning helps to prune data early. Imagine a MySQL table that is internally partitioned by month of the year, with one table for each month. Now, if a user requests data for the month of June, MySQL can quickly rule out all other partitions and only query the specific partition for the month of June. This is a significant performance benefit.
- Partitioning also helps a system scale horizontally. Each machine can store a specific set of partitions, and it won’t share anything with other machines. This is crucial to building a system that scales.
- Implementing partitioning and replication together is ideal for distributed systems—it achieves scalability and availability at the same time. The idea is to replicate each partition in multiple machines so that every partition is available even when one or a few of the machines are down. One replica of each partition is considered the leader, and the others are followers.

Learn about data partitioning in a distributed system.

Introduction

What Distributed Systems Achieve for Us

Data in Distributed Systems

Communication Between Nodes

Data Processing in Large Scale

Distributed System Architectural Patterns

Case Study 1: Apache Spark

Case Study 2: Apache Druid

Conclusion

Data Partitioning