Gain insights into Spark Java API, learn about data transformations and SQL operations, and discover how to integrate big data and Java for scalable, high-speed processing.

tarball.tar.gz

DataFrameBasicsRun

SparkDataframeExampleDB

This course serves as a comprehensive introduction to the Spark Java API. Experienced Java developers will use object-oriented programming (OOP) principles to apply theory to Apache Spark and big data practice.

You’ll learn the basic components and architecture of Spark, a leading framework for building big data applications, before implementing them in Java. You’ll also explore data transformations like grouping, sorting, and joining. Further, you’ll learn to support SQL operations in the database and create a big data and batch template application with Java.

By the end of the course, you’ll be familiar with Apache Spark and know how to integrate big data with Java environments through the Spark Java API. You’ll wrap up by learning about monitoring and support functions for a live Spark Java environment.

Combining a leading big data framework with a leading programming language, this course will empower you to work efficiently with large volumes of data, and process at scale and speed.

Mastering Big Data with Apache Spark and Java

In previous lessons, we used diagrams to slowly introduce to the reader how the Spark components interact, and how execution flow for a Spark application falls into place.

This lesson expand on this by presenting a broader picture of the Spark Landscape in a typical but straightforward application’s execution. It also dives into two of the most common cluster topologies in which a Spark application can run.


# The big picture

A diagram of the Spark ecosystem while executing a program is below:


The binding component between the application code that lives in the driver program and its execution on a cluster is usually a master node or the cluster manager. Which one is determined by how the cluster is set up and configured. Let’s expand on this a bit.


## Master or cluster manager?

Installing Spark on physical components (machines or servers), thus creating a Spark supported cluster, is usually done in two ways.

The resulting physical setup of the cluster defines what is usually referred to as the cluster mode, in which Spark applications are executed. It also imposes different constraints on various functionalities that the cluster exposes to Spark as an overall computing resource.


The following paragraphs explain the two most common modes.  




### Standalone mode

Although the **Standalone Mode** involves considerably more manual intervention while being set up and maintained, it is simpler to execute than the Cluster Mode discussed below. Standalone Mode is comprised of two groups of components: A single Master node and a group of Worker nodes.

The Master node is primarily a Resource Manager targeted at clusters expected to run Spark applications only because it greedily utilizes all the resources available when executing a Spark application.

Installation and cluster setup usually involves DevOps operations which are necessary to start all the types of nodes present in the cluster, either by using launch scripts or by manually placing a compiled version of Spark into each of the cluster nodes.


This is followed by the execution of different Spark distribution binaries required to start one node as the master, and the remaining nodes as workers.

Cluster configurations also have to be done manually through different configuration files and scripts.



### Cluster mode
Cluster Mode relies on a completely separate process outside the Spark distribution called cluster Manager, which acts as the resource manager and also incorporates extra and richer functionalities.

Managing a cluster consisting of potentially thousands of nodes is a herculean task, so the cluster manager was devised as a specific process specialized in the management and monitoring of resources available. It also allows other types of applications to run alongside Spark.

Furthermore, it provides good resource allocation, and tends to be installed by default on most cloud clusters available from different cloud providers.

Examples of cluster managers are both Hadoop YARN and Apache Mesos.

Regardless of the mode, when a cluster is fully set up, a driver program or Spark application can be configured to point to the address of the node that acts as the master or the cluster manager through the Spark Session or Spark Context,  The application can also be submitted to either of the previous components through Spark commands.


> **Note:** The physical layout and setup of a cluster, and the roles of each component, including where they might run as services or how to submit an application to the cluster, is expanded in the “[Deployment and Cluster Execution](https://www.educative.io/courses/mastering-big-data-apache-spark-java-api/local-and-cluster-based-execution)” chapter.

Installation and setup of a cluster that utilizes a cluster manager is a similar process to the one described before for standalone mode, when it comes to setting up the worker nodes. However, it differs considerably when setting up the cluster manager process itself.

> **Note:** Setting up a cluster from scratch falls outside the scope of this course, but you can find more resources here.


With both cluster modes summarized, let's dig into what a generic application’s flow would be.


## Application flow

The flow of the diagram shown before portrays a Spark application within the context of a general application execution flow.

The application’s code initiates the Spark execution in the driver program by creating a Spark session object where the SparkContext lives.

This is followed by a couple of operations. The first loads a file into a DataFrame, and the second applies a map transformation to the aforementioned file.

Both operations are chosen arbitrarily to relate to the previous lessons’ code, but they are interchangeable for any of the available repertoire of operations exposed through the Spark API.

Lastly, the `count()` action is invoked on the DataFrame, which triggers a cascade of events. Let’s analyze them separately by focusing on the main pieces or components of the diagram involved during execution.

### Spark Context

Although we have not explained or mentioned the Spark Context previously, it has a crucial role in the execution. According to the official Spark documentation definition, it is the:
___

"Main entry point for Spark functionality. A `SparkContext` represents the connection to a Spark cluster. It can be used to create RDDs and accumulators, and to broadcast variables on that cluster. "

___
In older versions of the Java Spark API, the de facto way of creating Datasets (RDDs or DataFrames) was done through the context. We have written code that does not rely on it to load, create, or manipulate datasets, but it can still be used to do so.

More importantly, the context allows the driver program to access the cluster’s resources through interaction with the Master node (if the cluster is configured in standalone mode) or through the cluster manager service (if the cluster is set up in cluster mode).

Fundamentally, Spark applications run as independent sets of processes on a cluster, coordinated by the `SparkContext` object in the driver program.

Moreover, the Spark Context holds the Spark configuration of the entire application. This has been dramatically simplified in Spark 3.0, the version that we are using through Maven. Thus, using the session for basic application configuration and bootstrapping should be enough in most situations.

That said, if an application needs to use older Spark dependencies’ versions, documentation for Spark 2.X SDK can be found in the [official spark docs.](https://spark.apache.org/docs/2.1.1/programming-guide.html)

> **Note:** We’ll cover additional programmatic interaction with the Spark context when we learn about  [Accumulators and Broadcast Variables](https://www.educative.io/courses/mastering-big-data-apache-spark-java-api/accumulators-and-broadcast-variables).

The diagram illustrates some crucial tasks that the context performs:

1. As the code flow reaches an action, it creates a Job consisting of Stages. These Stages are, in turn, split into Tasks.
2. Tasks of a Job are launched using a module in the Spark context called the task scheduler.




### The Driver Program

Also named the Spark driver or Spark application, the Drive Program contains the totality of the code base for a program.

In previous lessons’ diagrams, it was depicted sometimes as an element living outside or separately from the master node.

This is a logical separation in reality and, unless the program is running on a single server or pc, the driver program runs in tandem with the master node or the cluster manager depending on the cluster’s setup.

Because the driver program contains the Spark context, a cornerstone component of Spark distribution, it communicates with both the executor processes on the worker nodes and with the master node/cluster manager process for the duration of the application execution.


### Worker nodes

The worker nodes are essentially the worker bees of the cluster. They are contained within each node and their executor processes are spawned and assigned to tasks belonging to one or multiple jobs.


The nodes keep the data that needs to be processed in memory partitions that the executors have access to. If necessary, executors also use secondary storage, such as node disks or hard drives.



## Summary

This lesson provided an outlook of the main components involved during Spark execution and how they are linked to each other based on specific and well-defined roles on a cluster.

The dynamic of how a Spark application is run should rarely vary, at all, once a cluster is fully set up and ready to receive applications for execution. This allows the Spark developer to focus on the actual coding and usage of the Spark API to fulfil business requirements.


This lesson expands the previous landscape for a Spark Execution flow and Architecture.

Course Introduction

Spark Introduction and Basics

Getting Started with Spark

DataFrame Basic Operations

DataFrame Advanced Operations

Spark SQL and Other Functionalities

Building a Big Data Batch Application

Deployment and Cluster Execution

Monitoring and Performance Fundamentals

Conclusion

Apendix

Deep Dive: Internals of Spark Execution