Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

# Execution of a Spark Application

As discussed earlier, a Spark job is initiated when an action is performed. Internally, the SparkContext object's `runJob(...)` method is invoked. The call is passed on the scheduler. The scheduler runs as part of the driver and has two parts:

+ DAG Scheduler
+ Task Scheduler


The DAG scheduler breaks a job into a directed acyclic graph (DAG) of stages and constituent tasks. The task scheduler is responsible for submitting (shuffle or results) tasks from each stage to the cluster. The DAG scheduler is intelligent enough to skip stages for computing RDDs that have been persisted earlier in the same application. The DAG scheduler considers data locality when requesting placement preference for tasks. A task that acts upon a RDD partition residing on HDFS will have a placement preference for the datanode that hosts the particular RDD partition (node local). If the RDD is cached in memory then the task will be placed on the executor storing the RDD partition (process local).

The task scheduler receives a list of tasks and matches them with executors running for the application taking into account placement preferences. The tasks are assigned to free executors for execution. The executors are assigned tasks in the following preference:

+ Process-local tasks

+ Node-local tasks

+ Rack-local tasks

+ Arbitrary nonlocal tasks


Assigned tasks are launched through another component called the scheduler backend. The executor also reports back status updates to the driver program. If the task fails, it is resubmitted by the task scheduler onto another executor.

Before executing the task, the executor makes sure  jar and file  dependencies are up to date. Usually, these are cached from previous task runs. Next, the task code is deserialized and is received as part of the launch task message. Finally, the task code is executed in the same JVM as the executor. The result is serialized and sent back to the executor backend which forwards it to the driver as a status message update.


This lesson explains the role of the DAG and Task schedulers.

Execution of a Spark Application

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Execution of a Spark Application

Execution of a Spark Application