The Application Architecture Ecosystem

Spark in practice

We are at a point in the course where we can apply some of the concepts learned previously and work with a project closer to a real-world solution. This whole chapter of the course explains how to build a Spark-based, big data batch application. In this lesson, we describe its generic architecture, including the collaborating services that it interacts with.

Next, we delve into the application’s design and explain it in detail.

Once we are familiar with the design and architecture, we move to the code implementation; this last part also produces a template for a big data batch application that we can re-use and adapt to different business needs.

Let’s begin with the architecture.

Architecture

A possible generic architectural landscape for the batch application is illustrated in the following diagram:

The batch application resides, or rather is executed, in tier 4 and interacts with other components in adjacent tiers. To grasp how these components interact and work together as a cohesive solution, let’s explain in a bit more detail what functions and responsibilities each tier, and the components within them, have.

Note: This architecture does not intend by any means to be the most complete architecture possible or to address advanced architectural concerns such as scalability, availability, and more. It doesn’t address non-functional requirements like, for example, security. All these design considerations take quite more effort and working hours than this course covers. Nevertheless, the tiers and components depicted on it are a good foundation that can continually be expanded, enriched, and built upon.

1. Management and scheduling tier

The management and scheduling tier’s functionalities include, first, defining the batch jobs’ workflow. In other words, it defines the order, sequential or parallel or both, for jobs’ executions. It also encompasses actions to deal with failures or unexpected ...