Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

spark.tar.gz

SparkShellUI

SparkHistoryServerUI

Spark has come to dominate the big data processing space in a short span of time since its release and now serves as the de-facto unified big data processing engine in the industry. 

In this course, you will get a complete introduction to the basics of Spark. You will start by learning about the architecture, the application lifecycle, and its API.

From there, you will dive into the data frame data structure and its API as well as the strongly-typed datasets API. Lastly, you’ll get into the Spark SQL engine which will allow you to issue queries on structured data with a schema.

By the end of this course, you will have the confidence to use Spark in any of your big data projects.

An Introduction to Spark

# APIs
Spark offers APIs and data abstractions that significantly enhance the developer experience. The original Spark paper describes the low-level abstraction called 
 **Resilient Distributed Datasets (RDD)**. Later, others such as **DataFrames** and **Datasets** were added. Spark enables distributed data processing through functional transformations of collections of data (RDDs). The Spark API significantly reduces the size of programs compared to the size of the same programs when written in other frameworks such MapReduce. The Spark APIs are:


<ul>
<li>Resilient Distributed Datasets</li>

<li>DataFrames</li>

<li>Datasets</li>

</ul>

Get an introduction to the three Spark data abstractions - RDDs, DataFrames, and Datasets.

Spark Overview

DataFrames

Datasets

Spark SQL

Summary

Spark API

APIs