Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

# Disk Blocks & HDFS Blocks

We discussed a __disk block__ at the start of the chapter. It is the smallest unit of data that can be read from or written to a disk. Usually disk blocks are 512 bytes in size. The filesystem sitting on top of the physical disk works with an abstraction called the __filesystem block__, not with __disk blocks__. The filesystem block is often an integral multiple of disk blocks, usually a few kilobytes in size. However, this complexity is hidden from the end users of the filesystem.

HDFS is not a physical filesystem, but rather a virtual abstraction over distributed disk-based file systems. HDFS can't be browsed like the local filesystem. You need the __HDFS shell__, the __HDFS web UI__, or programmatic APIs to do that. The words __block__ and __blocksize__ have a different meaning in HDFS context. Let's explore them next.


# HDFS block

A file in HDFS is logically divided up into HDFS blocks. Each HDFS block is physically made of filesystem blocks of the underlying filesystem, which in turn is an integral multiple of the disk block size.

The benefit of block abstraction for a distributed file system like HDFS is that _a file can be larger than any single disk in the cluster_. In the latest version of Hadoop, HDFS has a  default block size of 128MB. However, if we store a 10MB file, it'll take up only 10MB of disk space, not 128MB. Storing a 1KB file in HDFS doesn't imply that on the disk a block of minimum size 128MB is written. HDFS, deals with disk blocks of a much smaller size. Therefore, the disk space used is actually 1KB for a 1KB file. The HDFS block of 128MB doesn't translate to a unit of storage. Instead, it is an abstraction to store the metadata in Namenode. It is the smallest unit the Namenode can reference in its memory. The underlying physical file system isn't divided into HDFS block-sized chunks. Let's explore another example. If we have three files of 10KB each, then the space consumed on disk would be __3 x 10KB = 30KB__ whereas the Namenode would hold 3 HDFS blocks, one per file in memory.

This lesson talks about the disk blocks, filesystems blocks and HDFS blocks.

Disk Blocks & HDFS Blocks

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Disk Blocks & HDFS Blocks

Disk Blocks & HDFS Blocks

HDFS block