Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

spark.tar.gz

SparkShellUI

SparkHistoryServerUI

Spark has come to dominate the big data processing space in a short span of time since its release and now serves as the de-facto unified big data processing engine in the industry. 

In this course, you will get a complete introduction to the basics of Spark. You will start by learning about the architecture, the application lifecycle, and its API.

From there, you will dive into the data frame data structure and its API as well as the strongly-typed datasets API. Lastly, you’ll get into the Spark SQL engine which will allow you to issue queries on structured data with a schema.

By the end of this course, you will have the confidence to use Spark in any of your big data projects.

An Introduction to Spark

## Application life cycle 
 
In this lesson, we'll track the various phases a Spark application goes through from the time it is submitted to when it is marked as completed.


1. As a user, the first step is to submit the Spark job to the cluster. Usually, this involves the user running the `spark-submit` command in a terminal window. The command spawns a process that talks to the cluster manager. If YARN is being used as the cluster management software, then the client process connects to the Resource Manager (RM) daemon. If the job is accepted, the RM will create the Spark driver process on one of the machines in the cluster.

2. Once the driver process starts running, it executes the user code. The code must establish a **SparkSession** which in turn sets up the ***Spark cluster***. The driver process and the executor processes are collectively referred to as the Spark cluster. We'll study SparkSession in-depth in upcoming lessons, but for now it is sufficient to know that SparkSession is a unified single point of entry to interact with underlying Spark functionality and allows for programming Spark with DataFrame and Dataset APIs. The SparkSession will talk to the cluster manager daemon, which will be RM in our case, to launch Spark executor processes on worker nodes.

3. The RM (cluster manager) will launch Spark executor processes on nodes across the cluster and return the location of executor processes to the driver process. The Spark cluster is setup at this point and the driver can communicate directly with the executor processes.

4. Once the Spark cluster is set up, the driver assigns tasks to executor processes and job execution begins. Data may be moved around and executors report on their status to the driver.

5. The driver exits when the Spark job completes and the cluster manager shuts down the executor processes on behalf of the driver. The cluster manager can be queried for the success or failure of the job.
 

## Demonstration
Now that we have learned more about Spark, let's see how it works in practice with an example. In this course, we'll be working with datasets related to Bollywood movies. The dataset can be downloaded for free from <a href="https://www.kaggle.com/mitesh58/bollywood-movie-dataset" target="_blank">Kaggle</a>. Execute the commands listed in the widget below and observe the results.

## Application life cycle 
 
In this lesson, we'll track the various phases a Spark application goes through from the time it is submitted to when it is marked as completed.


1. As a user, the first step is to submit the Spark job to the cluster. Usually, this involves the user running the `spark-submit` command in a terminal window. The command spawns a process that talks to the cluster manager. If YARN is being used as the cluster management software, then the client process connects to the Resource Manager (RM) daemon. If the job is accepted, the RM will create the Spark driver process on one of the machines in the cluster.

2. Once the driver process starts running, it executes the user code. The code must establish a **SparkSession** which in turn sets up the ***Spark cluster***. The driver process and the executor processes are collectively referred to as the Spark cluster. We'll study SparkSession in-depth in upcoming lessons, but for now it is sufficient to know that SparkSession is a unified single point of entry to interact with underlying Spark functionality and allows for programming Spark with DataFrame and Dataset APIs. The SparkSession will talk to the cluster manager daemon, which will be RM in our case, to launch Spark executor processes on worker nodes.

3. The RM (cluster manager) will launch Spark executor processes on nodes across the cluster and return the location of executor processes to the driver process. The Spark cluster is setup at this point and the driver can communicate directly with the executor processes.

4. Once the Spark cluster is set up, the driver assigns tasks to executor processes and job execution begins. Data may be moved around and executors report on their status to the driver.

5. The driver exits when the Spark job completes and the cluster manager shuts down the executor processes on behalf of the driver. The cluster manager can be queried for the success or failure of the job.
 

# Demonstration
Now that we have learned more about Spark, let's see how it works in practice with an example. In this course, we'll be working with datasets related to Bollywood movies. The dataset can be downloaded for free from <a href="https://www.kaggle.com/mitesh58/bollywood-movie-dataset" target="_blank">Kaggle</a>. Execute the commands listed in the widget below and observe the results.

Learn the various stages through which a Spark application goes through when executed by the Spark processing engine.

Spark Overview

DataFrames

Datasets

Spark SQL

Summary

Spark Application Lifecycle

Application life cycle