Spark Application Life Cycle

Spark Application Life Cycle

In this lesson, we’ll track the various phases a Spark application goes through, from the time submitted to when it’s marked completed.

1. As a user, the first step is to submit the Spark job to the cluster. Usually, this involves the user running the spark-submit command in a terminal window. The command spawns a process that talks to the cluster manager. If YARN is used as the cluster management software then the client process connects to the Resource Manager (RM) daemon. If the job is accepted, the RM will create the Spark driver process on one of the machines in the cluster.

2. Once the driver process starts running, it executes the user code. The code must establish a SparkSession which in sets-up the Spark cluster. The driver process and the executor processes are collectively referred to as the Spark cluster. We’ll study SparkSession in depth in upcoming lessons. For now, know that the SparkSession is a unified single point of entry to interact with underlying Spark functionality. It allows programming Spark with DataFrame and Dataset APIs. The SparkSession talks to the cluster manager daemon, in our case the RM, to launch Spark executor processes on worker nodes.

3. The RM (cluster manager) will launch Spark executor processes on nodes across the cluster and return the location of executor processes to the driver process. The Spark cluster is setup at this point and the driver can communicate directly with the executor processes.

4. Once the Spark cluster is ready, the driver assigns tasks to executor processes and job execution begins. Data may be moved around and executors report on their status to the driver.

5. The driver exits when the Spark job completes and the cluster manager shuts down the executor processes on behalf of the driver. The cluster manager can be queried for the success or failure of the job.