Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

spark.tar.gz

SparkShellUI

SparkHistoryServerUI

Spark has come to dominate the big data processing space in a short span of time since its release and now serves as the de-facto unified big data processing engine in the industry. 

In this course, you will get a complete introduction to the basics of Spark. You will start by learning about the architecture, the application lifecycle, and its API.

From there, you will dive into the data frame data structure and its API as well as the strongly-typed datasets API. Lastly, you’ll get into the Spark SQL engine which will allow you to issue queries on structured data with a schema.

By the end of this course, you will have the confidence to use Spark in any of your big data projects.

An Introduction to Spark

# Building blocks of a Spark application
In this lesson, we'll formally look at various components of a Spark application. A Spark application consists of one or several jobs. But a Spark job, unlike MapReduce, is much broader in scope. 

Each job is made of a directed acyclic graph of **stages**. A **stage** is roughly equivalent to a map or reduce phase in MapReduce. A stage is split into tasks by the Spark runtime and is executed in parallel on partitions of an RDD across the cluster. A task can be thought of as the single unit of work or execution that is sent to the Spark executor. Each task maps to a single core and works on a single partition of data. The relationship among these various concepts is depicted below:

A single Spark application can run one or more Spark jobs serially or in parallel. Cached RDDs output from one job can be made available to a second without requiring disk I/O in between. This makes certain computations extremely fast. A job always executes in the context of a Spark application. The spark-shell is an instance of a Spark application. 

Let's see an example to better understand jobs, stages and tasks. The example below creates two DataFrames, each consisting of integers from 0 to 9. Next, we transform one of the DataFrames to consist of multiples of three by multiplying each element by 3. Finally, we compute the intersection of the two DataFrames and sum up the result. The Scala code presented below can be executed in the terminal.

    // Start the Spark shell
    spark-shell

    // Create a dataframe consisting of ints from 0 to 9
    val dataFrame1 = spark.range(10).toDF()
    val dataFrame2 = spark.range(10).toDF()

    // Examine the number of partitions
    dataFrame1.rdd.getNumPartitions

    // Repartition the data
    val transformation1 = dataFrame1.repartition(20)
    val transformation2 = dataFrame1.repartition(20)

    // Create multiples of 3 from dataFrame1
    val mulsOf3 = transformation1.selectExpr("id * 3 as id")

    // Perform an inner join between the two dataframes.
    // transformation1.collect() = [0, 1, 2, 3, 4, 5. 6, 7, 8, 9]
    // mulsOf3.collect() = [0, 3, 6, 9, 12, 15, 18, 21, 24, 27]
    val join = mulsOf3.join(transformation2, "id")

    // Sum the intersection of the two dataFrames now
    // join.collect() = [0, 3, 6]
    val sum = join.selectExpr("sum(id)")

    // Perform an action to get the final result
    val result = sum.collect()

    // Examine the physical plan generated
    sum.explain()

# Transformations and actions

You need to draw a distinction between transformations and actions in a Spark job. All the statements before line 28 <code>sum.collect()</code> are transformations. A transformation changes a Spark data structure (such as a DataFrame) to a new one without altering the original data. Transformations are at the core of expressing your business logic using Spark. Remember, Spark works with immutable data-structures RDDs, which means we can't change the original RDD. We can generate a new one by applying a transformation to the elements of the existing RDD. However, transformations are lazily evaluated, meaning they are not performed until an action is performed on an RDD. Transformations get recorded or remembered as lineage. The delayed execution of transformations allows Spark to rearrange transformations for efficient execution. An action triggers a computation on an RDD and does something with the result. Unlike a transformation, an action is immediately evaluated. Our example on line 28 is the action that causes all previous transformations to be evaluated. There are two kinds of transformations:

Read-up on the constituents of a Spark application.

Spark Overview

DataFrames

Datasets

Spark SQL

Summary

Anatomy of a Spark Application

Building blocks of a Spark application

Transformations and actions