Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

# Testing Mapper

In the previous section, we wrote our `CarMapper` class. It takes in a long offset and a string line as input key-value pairs. The class then splits out car brand names and a count (always 1) as intermediate key-value pairs for the reducer function to consume. When developing code, we need to test it along the way. However, debugging and testing map reduce take a departure from the traditional testing and debugging of Java applications because of its distributed nature. We'll start with unit testing.

# Unit testing

Fortunately, the Java library __[MRUnit](https://mrunit.apache.org/)__ can be used to unit test MapReduce jobs. MRUnit has been retired; frameworks like Mockito are used instead to unit test MapReduce jobs. Nevertheless, we use MRUnit to get a feel for unit testing MapReduce. The unit test to exercise our `CarMapper` is shown below:

## `CarMapper` unit test


```java    
    @Test
    public void testMapper() throws IOException {

        MapDriver<LongWritable, Text, Text, IntWritable> driver =
                MapDriver.<LongWritable, Text, Text, IntWritable>newMapDriver()
                        .withMapper(new CarMapper())
                        .withInput(new LongWritable(0), new Text("BMW BMW Toyota"))
                        .withInput(new LongWritable(0), new Text("Rolls-Royce Honda Honda"))
                        .withOutput(new Text("bmw"), new IntWritable(1))
                        .withOutput(new Text("bmw"), new IntWritable(1))
                        .withOutput(new Text("toyota"), new IntWritable(1))
                        .withOutput(new Text("rolls-royce"), new IntWritable(1))
                        .withOutput(new Text("honda"), new IntWritable(2));

        driver.runTest();
    }
```

The above code is self-explanatory. The library uses the builder pattern to create an instance of a `MapDriver`, initialized with an instance of the `CarMapper` class. In the following lines, we specify the input key value pairs with the `withInput(...)` method and then specify the expected output using the `withOutput(...)` method.

The complete runnable unit test case appears in the code widget below:

This lesson demonstrates how to write a unit test case for a Mapper class.

Testing Mapper

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Testing Mapper

Testing Mapper