Gain insights into distributed systems, learn architectural patterns like replication and sharding, and explore Apache Spark and Druid to design and develop effective distributed applications.

data_eng.tar.gz

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

Apache_Druid

Mapreduce

aws_load_balancer

service_sharding

cqrs

apache_spark

apache_kafka

Gone are the days when monolithic applications were the norm. Most applications that we use today are too complex for that type of construction. Instead, most applications that appear as a single unit are actually built as a collection of inter-operating but independent computational components. Developers are expected to have mastery over a programming language or two. Moreover, they should also know the basics of distributed systems in order to succeed.

In this course, you’ll learn what a distributed system is, followed by the challenges unique to distributed systems. You’ll cover popular architectural patterns for achieving targets like replication and sharding. You’ll finish with a study of two popular distributed systems: Apache Spark and Apache Druid.

With the knowledge gained through this course, you’ll be able to design and develop distributed systems. You’ll be able to make optimal design and architectural choices based on the application requirements.

Introduction to Distributed Systems for Dummies

# Why do we need distributed systems?

We have already kind of answered this question in our previous lesson. But let’s look at the “why”  in detail.


There are quite a few reasons why we need distributed systems: 

- As you’ve seen in the last lesson, a non-distributed system runs on a single node, and that node always has a chance of crashing. Actually, nodes crash all the time in large systems, for example, at Google.
- The above means that it’s best to keep multiple copies of your server and database running on different nodes, and these nodes should be consistent in terms of functionality.
- With time, your system may grow. Say your My Cool App started with no user, and after six months it was so successful that you now have 1 million monthly active users. When you had no user, running the app on a single server made sense, but not after six months. Your server program needs to be running on possibly tens of machines now. Also, the database should have replicas to support read operations. More details will follow in the course.
- _Why do you need more nodes to run the server program?_ The answer is simple: hardware does have limitations. You can increase the RAM, CPU, and number of disks of your single node, but it will face a bottleneck eventually. This is why supercomputers with thousand cores and terabytes of RAM are not very common.
- Running the server program on multiple nodes ensures that if one or more of the nodes fail, your system is still functioning (at least partially). It also ensures no one node is overwhelmed with too many requests due to resource exhaustion (remember that machines have limited resources) and eventually broken.

Numerous systems in the world right now are built as distributed systems. And with the emergence of cloud technologies in the last decade, it is far easier to build systems as distributed ones.

> **Note**: Though it may be easier to create a distributed system now, it’s still a difficult thing to achieve. You should need to evaluate carefully whether you actually need a distributed system.

# Evaluating the need for a distributed system

As a system owner, you should be very careful about opting for a distributed system. You should always evaluate what your users’ needs are and whether they can be properly served without implementing a distributed system. It might sound cool to build a system like this for your cool startup, but it may also not be worth the effort.

For example, in our My Cool App, we could evaluate the following to decide what to do:

- Did we need to add more nodes for the server program? Or would a reasonably larger node (in terms of resources) be enough? What does our traffic look like? For example, say initially our single node had 20 GB of RAM, maybe we could have gone for 32 GB and that would have been enough?

- To handle failed node scenarios in server nodes, could it be enough to have one or two standby nodes to take over once there is a failure? Yes, but that means there would be downtime in our system. But what if a few minutes of downtime does not make any difference in our system?
- Was it absolutely necessary to create a multi node database system? Could we have one database node to handle all the traffic, and would that be enough?
- When there is a crash in the database node, would it be possible to spin up a new database node and copy all the data? Maybe from a backup? Would the downtime affect our business?
- Is the cost of building and running a distributed system worth it, given our business requirements and growth?

The bottom line is be cautious before making a distributed system. Find out whether it is worth it to go for the complexity of a distributed system. And go for it if you have asked all the right questions and the answer is still that a distributed system would be best, go for it!.

In this course, we will build a solid understanding of distributed systems so that when the time comes, you can ask yourself critical questions like these and answer them confidently.

# Key takeaways

- Hardware is failure prone. With time, the performance of hardware will degrade. Nothing lasts forever, and neither does hardware.
- Having multiple copies of the same program allows the system to continue to function, at least partially.
- Distributed systems may seem alluring to implement but before deciding to go for it, careful evaluation of business requirements and use-case analysis is a must.

Understand why distributed systems matter.

Introduction

What Distributed Systems Achieve for Us

Data in Distributed Systems

Communication Between Nodes

Data Processing in Large Scale

Distributed System Architectural Patterns

Case Study 1: Apache Spark

Case Study 2: Apache Druid

Conclusion

The Why of Distributed Systems

Why do we need distributed systems?