Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

## Distcp

Distributed Copy tool, also known as __distcp__, is one of the important tools of Hadoop. Commonly used in the industry for moving data around, it is as an example of a problem that MapReduce can solve. The Distcp tool allows for parallel processing of files on the same Hadoop cluster or between two Hadoop clusters. It can copy files or directories. Distcp is implemented as a map reduce job with no reduce phase. The mappers run in parallel across the cluster to perform the copy. This reduces the time required to copy the same data sequentially. Each file is copied by one map task; the smallest unit of work for Distcp is a file. If the number of mappers is set to one, the lone mapper writes one replica of each block of a file on the running node, and the second and third replicas spreads out across the cluster. This creates an imbalance where the node running the map task will hold all the copy data until its disk becomes full.

By default, Distcp tries to assign each map task a fixed set of files, so each map task copies an equal number of bytes. However, since a file is copied by one map task, it is impossible to split-up a single file among different mappers to copy. Therefore, the load for each map task may not be exactly even. You can use a dynamic strategy for copying, where the files are divided into several buckets whose number is more than the number of mappers. Each bucket is processed by a map task and if any map task finishes its bucket early, it can pick up one of the remaining buckets. The strategy can be specified as a command line argument.


# Distcp

Distributed Copy tool, also known as __distcp__, is one of the important tools of Hadoop. Commonly used in the industry for moving data around, it is as an example of a problem that MapReduce can solve. The Distcp tool allows for parallel processing of files on the same Hadoop cluster or between two Hadoop clusters. It can copy files or directories. Distcp is implemented as a map reduce job with no reduce phase. The mappers run in parallel across the cluster to perform the copy. This reduces the time required to copy the same data sequentially. Each file is copied by one map task; the smallest unit of work for Distcp is a file. If the number of mappers is set to one, the lone mapper writes one replica of each block of a file on the running node, and the second and third replicas spreads out across the cluster. This creates an imbalance where the node running the map task will hold all the copy data until its disk becomes full.

By default, Distcp tries to assign each map task a fixed set of files, so each map task copies an equal number of bytes. However, since a file is copied by one map task, it is impossible to split-up a single file among different mappers to copy. Therefore, the load for each map task may not be exactly even. You can use a dynamic strategy for copying, where the files are divided into several buckets whose number is more than the number of mappers. Each bucket is processed by a map task and if any map task finishes its bucket early, it can pick up one of the remaining buckets. The strategy can be specified as a command line argument.


This lesson talks about the distributed copy tool.

Distcp

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Distcp

Distcp