Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

# High Availability

High availability is characteristic of a distributed system. It is defined as the ability of a system or system component to be continuously operational for a long period of time. For example,  Amazon's ubiquitous S3 storage boasts a [99.99% availability](https://aws.amazon.com/s3/storage-classes/) over a given year.

To achieve high availability for HDFS, we need more than one instance of the Namenode to avoid downtime and failures during software/hardware upgrades . In HA setup, one Namenode serves client queries and is known as the Active Namenode. The rest are known as standby Namenodes. If the active Namenode experiences a failure, a standby Namenodes takes over.

**Working**

Imagine a cluster with two Namenodes. In order for the standby Namenode to successfully take over incase of failure, it exactly imitates the actions taken by the active Namenode on its namespace state. This is done by deploying __JournalNodes__. Like a _journal_, the JournalNodes keep a record of all the changes the active Namenode makes on its namespace. Because this is a distributed system, the changes are recorded to a majority of the JournalNodes. We need more than one JournalNode to record Namenode's activities because JournalNodes themselves are prone to failure.

How do we define the majority in case of an even number of JournalNodes? We don't! We must run an odd number of JournalNodes starting with at least three. With three nodes, the active Namenode writes the modifications it makes to its namespace to two out of the three JournalNodes. This allows for tolerating a single machine failure. The following formula dictates the number of failures a JournalNode will tolerate:

> $$\frac{N-1}{2}$$

where _N_ is the number of JournalNodes in HA setup. Plug in various numbers and see the number of JournalNodes that can go down before rendering the system unusable.

The illustration below shows a typical HDFS set-up with JournalNodes. There's also Zookeeper (discussed later) and Failover Controller abbreviated as ZK and FC respectively in the illustration. The FC is a Zookeeper related component that periodically pings the Namenode for health-checks and reports any failures back to Zookeeper.

This lesson explains how high availability is implemented for HDFS.

High Availability

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

High Availability

High Availability