Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

# Block Replication

Before discussing what makes HDFS fault-tolerant and highly available, let's first understand the term fault tolerance. Wikipedia defines fault tolerance as the property that enables a system to continue operating properly in the event of the failure of some of its components. Said another way, _fault tolerant computing_ is a form of full hardware redundancy. Two (or more) systems operate in tandem, mirroring identical applications and executing instructions in lock step. When a hardware failure occurs in the primary system, the secondary system running an identical application simultaneously takes over with no loss of service and zero downtime.

The secret sauce behind HDFS's ability to withstand corrupted or lost data is the replication of data blocks. If a file comprises of 4 HDFS blocks and the replication factor is 3, then each data block would have 3 copies of itself. These copies spread out in the cluster on physically separate machines, are a total of 12 blocks. Replication ensures that if one data block becomes corrupted or hardware failure occurs, then the read request can still be fulfilled by another available clone of the block. This setup allows for __self-healing__.  A lost block due to corruption or machine failure can be replicated to other live machines by making a copy of the healthy clone. The replication factor is controlled by the property `dfs.replication`. Some applications use a higher replication factor for the data blocks of a popular file to better distribute the read load across the cluster.

# Selection of nodes for replication

When it comes to the placement of replicated data blocks in a cluster, there are a spectrum of possibilities. On one end of the spectrum, all three data blocks can be placed on one node. Or all three can be placed in separate data-centers. In the former case, the write bandwidth is minimized because the replicas are written to the same node. No redundancy is realized because if the single node goes down, all replicas are lost. In the latter case, redundancy is maximized at the cost of   bandwidth.

In practice, Hadoop places the first replica on the  same node as the client. If the client is runnning outside the cluster, a node is chosen at random. The second replica is placed on a randomly chosen rack different than the first replica. The third replica is placed on a randomly chosen node on the same rack as the second replica. Any further replicas are placed on randomly selected nodes without placing too many replicas in the same rack.

An pictorial depiction of how a block gets replicated on a two rack cluster with eight machines is shown below:

This lesson discusses how HDFS replicates data blocks across the cluster for redundancy.

Block Replication

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Block Replication

Block Replication