Gain insights into Spark Java API, learn about data transformations and SQL operations, and discover how to integrate big data and Java for scalable, high-speed processing.

tarball.tar.gz

DataFrameBasicsRun

SparkDataframeExampleDB

This course serves as a comprehensive introduction to the Spark Java API. Experienced Java developers will use object-oriented programming (OOP) principles to apply theory to Apache Spark and big data practice.

You’ll learn the basic components and architecture of Spark, a leading framework for building big data applications, before implementing them in Java. You’ll also explore data transformations like grouping, sorting, and joining. Further, you’ll learn to support SQL operations in the database and create a big data and batch template application with Java.

By the end of the course, you’ll be familiar with Apache Spark and know how to integrate big data with Java environments through the Spark Java API. You’ll wrap up by learning about monitoring and support functions for a live Spark Java environment.

Combining a leading big data framework with a leading programming language, this course will empower you to work efficiently with large volumes of data, and process at scale and speed.

Mastering Big Data with Apache Spark and Java

# Core components

Behind the scenes, Spark is comprised of a core component on top of which different libraries sit. This is no accident, as the creators of Spark applied this type of architecture to continue adding modules pertaining to different functionalities.

This type of architecture resembles a “Plugin Architecture,” in which features can be developed and incorporated over time.

Let’s take a brief look at each of them.


## Spark Core

The nucleus of Spark contains the basic but fundamental functionalities for scheduling applications execution, memory management, storage systems’ interaction, fault recovery, etc.

**Spark Core** is the home of the **Resilient Distributed Dataset (RDD)** data structure, an in-memory fault-tolerant and immutable collection of elements representing partitioned data. Besides raw data, it can also contain a more complex type of data such as Scala, Python, or Java programmatic constructs (such as classes or data structures.)

RDDs have been a part of Spark since the 1.0 version and allow object-based and functional approaches to manipulate the data it represents.

This translates into more complex, richer, data manipulation by having functions applied to the elements of collections. A typical example of this is the aptly named **Pair RDD**: a representation of a collection of data identified by a key or ID for each of the records.

The following image depicts this extensively used technique in the early days of Spark, and is used frequently nowadays in Spark Scala API, a library with a stronger focus on the functional programming paradigm:





The data structure used in this course, and commonly used as the de-facto abstraction provided by the Spark Java API, resides in a different Spark component. Let’s take a quick look.

## Spark SQL

Spark SQL is a component that works with data in a structured fashion as well as allows data querying in a style that is quite similar to Structured Query Language (SQL). The crucial difference here is the possibility of querying distributed information efficiently.

Spark SQL allows the Developer to work with metadata about the structure of the DataFrame, such as columns types and the amount of rows, It then provides information as to the nature of the computations that can be executed on these abstractions.

Moreover, Spark’s SQL engine includes the following added features: optimization of both queries and transformations applied to the collections of data, queries’ fault tolerance, and support for various DataSources such as Avro, Parquet, JSON, JDBC, etc.


> Interaction with these DataSources are covered in a separate [chapter](https://www.educative.io/courses/mastering-big-data-apache-spark-java-api/ingesting-databases).

To encapsulate these added functionalities another abstraction is available on the API: the DataFrame.

The DataFrame, introduced in Spark version 1.3, is the leading actor throughout this course, so we dedicate plenty of lessons to learning about it, mastering its use, and reaping its benefits. For now, we can briefly describe what differentiates it from the RDD.



### Comparison of RDDs and DataFrames

Like RDDs, DataFrames also represent a distributed collection of elements, but it incorporates a representation of the data into named columns and rows.

RDDs are designed to work with both structured and unstructured data. However, the DataFrame was built to work primarily with structured and semi-structured data by exposing a schema.

A schema includes metadata about columns and their types, which allows for easier organization and manipulation of the data by offering a table-like representation of the DataFrame.

DataFrames also incorporate functionality to infer the schema of raw ingested or manipulated data, and the possibility of manually defining or even redefining it.

Lastly, DataFrames can be debugged while the RDDs did not provide this capability.


## Spark streaming

Spark streaming is the component that allows for processing of live streams of data being ingested and processed as it arrives. An example could be the log’s analysis or messages coming from a queue.

The key factor of this module is that it provides functionality to process information in almost real time.

Spark streaming allows for highly scalable, fault-tolerant processing of streams to support high throughputs. It also supports technologies such as Apache Kafka or AWS Kinesis, and even lower-level inputs such as TCP Sockets.

The key to understanding how streaming works internally is the concept of micro batching, which groups live data into relatively small batches of information that are then processed one after the other.

Some operations to transform the data within a stream are similar to those available for DataFrames, whereas other operations only apply to streams.

>**Note:** Spark streaming is not taught in this course. However, the following 
 [Spark Streaming Official Documentation](https://spark.apache.org/docs/latest/streaming-programming-guide.html) is be a good starting point for the interested reader.




## MLib and GraphX components

**MLib**, as its name implies, is a library that includes standard machine learning functionality, whereas **GraphX** is a library to process and manipulate graphs which allows a developer to perform computations in parallel.

The machine learning library shipped with Spark already contains implementations of various machine learning algorithms, such as clustering, classification, or regression, among others.

On the other hand, the GraphX API introduces the graph abstraction, a directed graph with properties attached to both vertices and edges, based on the RDD. It also includes typical graph processing algorithms for joining vertices, working with sub-graphs, etc.
 
> **Note:** Sadly, these two components are not taught in this course, as they also require a solid understanding of machine learning, graph theory, and algorithms realms well beyond our course’s scope.
 

# Cluster managers

If there's anything that Spark excels at,  it’s the ability to efficiently scale processing from one to potentially thousands of compute nodes.

To achieve this, Spark can run over different types of distributed cluster managers, including Hadoop YARN, Apache Mesos, and a standalone cluster manager.

Fundamentally, a **cluster manager** is a service or process that works as an external service for acquiring and managing resources in a cluster.

When initially installing and running simple tasks, for example during development stages, the standalone cluster manager allows for execution on a set of machines or even a simple PC (where it utilizes the PC’s cores as nodes).

For real-life scenarios, a cluster manager such as Hadoop or AWS EMR should be the preference. These can be located either on-premise (Hadoop) or in the cloud (AWS EMR). It makes no difference to the manager when it comes to managing the cluster.

How does Spark work with a cluster? A glance at its runtime architecture can shed some light on its inner workings.

# Spark runtime architecture

Spark was built to use a **master-workers architecture**. This means that a central node acts as the resource coordinator for the duration of processing that takes place while running a Spark application.

In the Spark landscape, thd central node acting as the master component is named the “cluster manager service,” or simply the “cluster manager.” A simple image of all these parts and their connections are illustrated in the following diagram:


Get introduced to Spark main components and cluster managers.

Course Introduction

Spark Introduction and Basics

Getting Started with Spark

DataFrame Basic Operations

DataFrame Advanced Operations

Spark SQL and Other Functionalities

Building a Big Data Batch Application

Deployment and Cluster Execution

Monitoring and Performance Fundamentals

Conclusion

Apendix

Components and Architecture

Core components

Spark Core