Gain insights into distributed systems, learn architectural patterns like replication and sharding, and explore Apache Spark and Druid to design and develop effective distributed applications.

data_eng.tar.gz

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

Apache_Druid

Mapreduce

aws_load_balancer

service_sharding

cqrs

apache_spark

apache_kafka

Gone are the days when monolithic applications were the norm. Most applications that we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems in order to succeed.

In this course, you’ll learn what a distributed system is, followed by the challenges unique to distributed systems. You’ll cover popular architectural patterns for achieving targets like replication and sharding. You’ll finish with a study of two popular distributed systems: Apache Spark and Apache Druid.

With the knowledge gained through this course, you’ll be able to design and develop distributed systems. You’ll be able to make optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

**(True or False)** A system built on $3$ nodes targeting 100 concurrent users can easily handle 10k concurrent users.

(True or False) A system built on <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>3</mn></mrow><annotation encoding="application/x-tex">3</annotation></semantics></math>3 nodes targeting 100 concurrent users can easily handle 10k concurrent users.

Scalability for databases is to the capability of handling an increased amount of data and queries.

Scaling up means adding more nodes to the system.

Horizontal scaling can be cheaper than vertical scaling for many systems.

The nodes after horizontal scaling are totally independent of one another, regardless of what type of system we have.

**(Select all that apply)** Which of these statements are correct?

(Select all that apply) Which of these statements are correct?

Randomly forwarding requests to any server node.

Forwarding to servers that have even IDs.

Tracking the least number of connections in a node and forwarding to the one with the least connections.

Which one is an example of load balancing in a distributed system?

Which one is an example of load balancing in a distributed system?

**(True or False)** Randomized load balancing will work in all different systems.

(True or False) Randomized load balancing will work in all different systems.

**(True or False)** A load balancer does not need to evenly distribute the requests to server nodes.

(True or False) A load balancer does not need to evenly distribute the requests to server nodes.

If a node is down, remove it from the system and never check whether it's back.

Stop forwarding requests to the faulty node.

Initiate the process of spinning up a new node.

What can a load balancer do to achieve availability in a distributed system?

What can a load balancer do to achieve availability in a distributed system?

**(True or False)** While using QPS as a system load metric, it’s not necessary to consider the time of  day.

(True or False) While using QPS as a system load metric, it’s not necessary to consider the time of day.

Write half of the data in one node and the other half in a different node.

Delete some data so that node can use more resources into serving queries.

How should we scale a read-heavy system?

**(True or False)** $p80$ is $20ms$ means _the system responds to $20%$ of the user requests in $20ms$.

(True or False) <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mn>80</mn></mrow><annotation encoding="application/x-tex">p80</annotation></semantics></math>p80 is <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>20</mn><mi>m</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">20ms</annotation></semantics></math>20ms means _the system responds to <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>20</mn></mrow><annotation encoding="application/x-tex">20%</annotation></semantics></math>20 of the user requests in <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>20</mn><mi>m</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">20ms</annotation></semantics></math>20ms.

Evaluate your understanding of scalability and load balancers.

Introduction

What Distributed Systems Achieve for Us

Data in Distributed Systems

Communication Between Nodes

Data Processing in Large Scale

Distributed System Architectural Patterns

Case Study 1: Apache Spark

Case Study 2: Apache Druid

Conclusion

Quick Quiz #4: Scalability and Load Balancers