Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

# Testing MapReduce

So far, we have learned how to write mapper and reducer classes and their corresponding unit tests. But ideally, we want to test our MapReduce job end to end. There are different ways to run a MapReduce job:


+ Using the `ToolRunner` class to run a MapReduce job on a local machine. The MapReduce job must implement the interface `Tool`. This doesn't require any running Hadoop daemons.

+ Setup a Hadoop cluster on a local machine in __pseudo distributed mode__ and then submit the job to the cluster.

+ Submit the MapReduce job to an actual cluster consisting of many machines.

We'll start with the first option by writing a program that implements the `Tool` interface. However, any class implementing `Tool` interface must also implement the interface `Configurable`, which is in turn extended by `Tool`. The easiest way for a MapReduce job is to derive from one of the Hadoop's helper class `Configured` which already implements the interface `Configurable`. If we name our MapReduce job  the class signature would look as follows:

```java
public class CarCounterMrProgram extends Configured implements Tool {
    // ... class body
}
```

The input to the program will live on the local disk. Create an object of type `Configuration`. Specify this information, along with  setting `mapreduce.framework.name`, to local. These changes are shown on lines 27 and 28 in the class `CarCounterMrProgram` below in the code widget. The class `CarCounterMrProgram` also represents our MapReduce job. We carry over the mapper and reducer classes created in the previous sections without any changes. We also create a class `CarMRInputGenerator` to generate random data. Read the comments in the code widget below and examine the various classes. Unfortunately, the code is not runnable because of hostname resolution challenges in a VM, required for testing the MapReduce program.


In this lesson, we'll demonstrate how to test a MapReduce program on a local machine.

Testing MapReduce Program

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Testing MapReduce Program

Testing MapReduce