Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

spark.tar.gz

SparkShellUI

SparkHistoryServerUI

Spark has come to dominate the big data processing space in a short span of time since its release and now serves as the de-facto unified big data processing engine in the industry. 

In this course, you will get a complete introduction to the basics of Spark. You will start by learning about the architecture, the application lifecycle, and its API.

From there, you will dive into the data frame data structure and its API as well as the strongly-typed datasets API. Lastly, you’ll get into the Spark SQL engine which will allow you to issue queries on structured data with a schema.

By the end of this course, you will have the confidence to use Spark in any of your big data projects.

An Introduction to Spark

## Execution of a Spark application


As discussed earlier, a Spark job is initiated when an action is performed. Internally, this invokes the SparkContext object's <code>runJob(...)</code> method. The call is passed on the scheduler. The scheduler runs as part of the driver and has two parts:

<ul>
<li>DAG Scheduler</li>
<li>Task Scheduler</li>
</ul>

# Execution of a Spark application


As discussed earlier, a Spark job is initiated when an action is performed. Internally, this invokes the SparkContext object's <code>runJob(...)</code> method. The call is passed on the scheduler. The scheduler runs as part of the driver and has two parts:

<ul>
<li>DAG Scheduler</li>
<li>Task Scheduler</li>
</ul>

Get familiarity with the role of the DAG and Task schedulers in the context of Spark.

Spark Overview

DataFrames

Datasets

Spark SQL

Summary

Execution of a Spark Application

Execution of a Spark application