Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

# Avro IDL & RPC

Other serialization formats such as Thrift, Protocol Buffers, and others provide an Interface Definition Language(IDL). IDL allows the user to express a schema similar to you can write code in a programming language, rather than using JSON to specify the schema. Avro offers an equivalent: Avro IDL.

The Avro IDL comes with its own rules and syntax. It appear similar to popular languages like Java, C++, and Python. We'll present a simple example demonstrating the use of IDL. More details can be found [here](https://avro.apache.org/docs/current/idl.html). Each Avro IDL file defines a single Avro Protocol. When compiled, the output is a JSON format Avro Protocol file with extension _.avpr_. We can represent a car record in IDL format as follows:

```text
/**
 * Car schema expressed in Avro IDL as a protocol
 */
@namespace("io.datajek")
protocol CarProtocol {

  record Car {
    string make;
    string model;
    int year;
    int horsepower;
  }

string carToString(Car car);
}
```
Once we define the _.avdl_ file, we can convert it into an _.avpr_ file like this:

```bash
java -jar avro-tools-1.9.1.jar idl carProtocol.avdl carProtocol.avpr
```
The protocol file produced has this content:
```text
{
  "protocol" : "CarProtocol",
  "namespace" : "io.datajek",
  "doc" : "* Car schema expressed as Avro IDL",
  "types" : [ {
    "type" : "record",
    "name" : "Car",
    "fields" : [ {
      "name" : "make",
      "type" : "string"
    }, {
      "name" : "model",
      "type" : "string"
    }, {
      "name" : "year",
      "type" : "int"
    }, {
      "name" : "horsepower",
      "type" : "int"
    } ]
  } ],
  "messages" : {
    "carToString" : {
      "request" : [ {
        "name" : "car",
        "type" : "Car"
      } ],
      "response" : "string"
    }
  }
}
```
From the protocol file produced, we can generate the Java class `Car` as follows:

```bash
java -jar avro-tools-1.9.1.jar compile protocol carProtocol.avpr .
```

There's an interface `CarProtocol` generated along with the class `Car`.


This may get confusing, given the many ways you can work with Avro. The following diagram shows the possible work-flows with Avro. Start with an _.avdl_ file and auto-generate code. Or you could start with a _.avsc_ file and auto-generate code. The final option is to skip auto-generated code and choose to work directly with the _.avsc_ file.


This lesson explains the Avro Interface Definition Language and Avro RPC.

Avro: IDL & RPC

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Avro: IDL & RPC

Avro IDL & RPC