Lambda Architecture

In this chapter, the two patterns we have seen so far are pretty simple in terms of conceptual understanding. In this lesson, we will explore a more complicated architectural pattern called lambda architecture. It is an architecture suitable for many data-heavy systems.

Let’s dive in.

Lambda architecture overview

Like many of our previous lessons, we will discuss architecture with an example. Let’s look at the following scenario.

• We are building the backend system for an IoT device producing company with a large user base.

• The company needs real-time analytics on the performance and usage of its devices. But it’s okay to have a small fraction of inaccuracy in this case.

• On the other hand, the company also requires 100% accurate analytics on the data. But this requirement does not have to be fulfilled in real-time. In fact, for this use case, generating daily or weekly reports is sufficient. But note that there should not be any error in the reports for any reason, whether due to data duplication or data loss.

• From the client side, every few seconds, the devices send usage and performance data to the backend system. The data comes as events.

• The company has millions of customers. As a result, millions of devices transmit millions of events every second.

Given the above scenario, one possible way to design the system is to implement the lambda architecture.

Note: In the lambda architecture, events are processed both in real-time and in batch. The real-time path ensures speed, whereas the batch path ensures correctness.