Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

spark.tar.gz

SparkShellUI

SparkHistoryServerUI

Spark has come to dominate the big data processing space in a short span of time since its release and now serves as the de-facto unified big data processing engine in the industry. 

In this course, you will get a complete introduction to the basics of Spark. You will start by learning about the architecture, the application lifecycle, and its API.

From there, you will dive into the data frame data structure and its API as well as the strongly-typed datasets API. Lastly, you’ll get into the Spark SQL engine which will allow you to issue queries on structured data with a schema.

By the end of this course, you will have the confidence to use Spark in any of your big data projects.

An Introduction to Spark

## Building blocks of a Spark application
In this lesson, we'll formally look at various components of a Spark application. A Spark application consists of one or several jobs. But a Spark job, unlike MapReduce, is much broader in scope. 

Each job is made of a directed acyclic graph of **stages**. A **stage** is roughly equivalent to a map or reduce phase in MapReduce. A stage is split into tasks by the Spark runtime and is executed in parallel on partitions of an RDD across the cluster. A task can be thought of as the single unit of work or execution that is sent to the Spark executor. Each task maps to a single core and works on a single partition of data. The relationship among these various concepts is depicted below:

# Building blocks of a Spark application
In this lesson, we'll formally look at various components of a Spark application. A Spark application consists of one or several jobs. But a Spark job, unlike MapReduce, is much broader in scope. 

Each job is made of a directed acyclic graph of **stages**. A **stage** is roughly equivalent to a map or reduce phase in MapReduce. A stage is split into tasks by the Spark runtime and is executed in parallel on partitions of an RDD across the cluster. A task can be thought of as the single unit of work or execution that is sent to the Spark executor. Each task maps to a single core and works on a single partition of data. The relationship among these various concepts is depicted below:

Read-up on the constituents of a Spark application.

Spark Overview

DataFrames

Datasets

Spark SQL

Summary

Anatomy of a Spark Application

Building blocks of a Spark application