Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

# Avro with Code Generation

The previous lesson demonstrated a program that used `GenericRecord` interface to serialize and deserialize data. This approach requires us to cast the individual fields of the record to the right data-type. For instance, if we want to print the model year of each car record, the program would look like as this:

import org.apache.avro.file.DataFileReader;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.io.DatumReader;
import org.apache.avro.Schema;

import java.io.File;
import java.io.IOException;

public class AvroReadExample {

    public static void main(String[] args) throws IOException {
        (new AvroReadExample()).deserializeAvroFile();
    }

    public void deserializeAvroFile() throws IOException {
        DatumReader<GenericRecord> datumReader = new GenericDatumReader<>();
        DataFileReader<GenericRecord> dataFileReader = new DataFileReader<>(new File("./fancyCars.avro"), datumReader);

        System.out.println();
        System.out.println("Car Years :");
        GenericRecord record = null;
        while (dataFileReader.hasNext()) {
            record = dataFileReader.next(record);
            int carYear = (int) record.get("year");
            System.out.println(carYear);
        }
    }
}


In __line 25__, we explicitly cast the object returned from the `get()` method to an integer. The code will not compile otherwise. Another option Avro provides is to compile the schema into Java classes that mimic the schema definition and can then be used in the program. The need for casts or reading the schema is thus eliminated. There are two ways to compile the schema into Java classes:

+ The first is to use the avro-tools jar file to compile the schema. We'll demonstrate this approach with our car example.

+ The second method is to use a maven plugin that automatically compiles the schema. This requires adding the specific maven plugin to the project pom file. More details are read [here](https://avro.apache.org/docs/1.8.2/gettingstartedjava.html#Compiling+the+schema). Under the hood, the plugin uses avro-tools jar, so we do not cover it here.

We can generate the class `Car` from the schema file car.avsc as follows:

```bash
java -jar /path/to/avro-tools-1.9.1.jar compile schema <schema file> <destination>

java -jar avro-tools-1.9.1.jar compile schema car.avsc .
```

Execute the above command by connecting to the terminal below and changing directory to DataJek. If you execute this command in the terminal the schema file has the value __io.datajek__ for the attribute __namespace__. This causes the auto-generated Java class `Car` to be created under the directory `io/datajek/Car.java`. If we set the namespace attribute to empty string, then the `Car` class is produced in the same directory from where we execute the compile command.

This lesson demonstrates how to work with Avro while using generated code.

Avro: Code Generation

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Avro: Code Generation

Avro with Code Generation