Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

spark.tar.gz

SparkShellUI

SparkHistoryServerUI

Spark has come to dominate the big data processing space in a short span of time since its release and now serves as the de-facto unified big data processing engine in the industry. 

In this course, you will get a complete introduction to the basics of Spark. You will start by learning about the architecture, the application lifecycle, and its API.

From there, you will dive into the data frame data structure and its API as well as the strongly-typed datasets API. Lastly, you’ll get into the Spark SQL engine which will allow you to issue queries on structured data with a schema.

By the end of this course, you will have the confidence to use Spark in any of your big data projects.

An Introduction to Spark

# Execution of a Spark application


As discussed earlier, a Spark job is initiated when an action is performed. Internally, this invokes the SparkContext object's <code>runJob(...)</code> method. The call is passed on the scheduler. The scheduler runs as part of the driver and has two parts:

<ul>
<li>DAG Scheduler</li>
<li>Task Scheduler</li>
</ul>

The DAG scheduler breaks a job into a directed acyclic graph (DAG) of stages and constituent tasks. The task scheduler is responsible for submitting (shuffle or results) tasks from each stage to the cluster. The DAG scheduler is intelligent enough to skip stages for computing RDDs that have been persisted earlier in the same application. The DAG scheduler considers data locality when requesting placement preference for tasks. A task that acts upon a RDD partition residing on HDFS will have a placement preference for the datanode that hosts the particular RDD partition (node local). If the RDD is cached in memory then the task will be placed on the executor storing the RDD partition (process local).

The task scheduler receives a list of tasks and matches them with executors running for the application, taking placement preferences into account. The tasks are then assigned to free executors for execution. The executors are assigned tasks in the following preference:

<ol>
<li>Process-local tasks</li>

<li>Node-local tasks</li>

<li>Rack-local tasks</li>

<li>Arbitrary nonlocal tasks</li>
</ol> 


Assigned tasks are launched through another component called the **scheduler backend**. The executor also reports back status updates to the driver program. If the task fails, it is resubmitted by the task scheduler onto another executor.

Before executing the task, the executor makes sure jar and file dependencies are up to date. Usually, these are cached from previous task runs. Next, the task code is deserialized and is received as part of the launch task message. Finally, the task code is executed in the same JVM as the executor. The result is serialized and sent back to the executor backend which forwards it to the driver as a status message update.

Get familiarity with the role of the DAG and Task schedulers in the context of Spark.

Spark Overview

DataFrames

Datasets

Spark SQL

Summary

Execution of a Spark Application

Execution of a Spark application