Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

## Avro

Avro is a data serialization system that allows for remote procedure calls and the fast and compact serialization of data. The defining feature of Avro is a schema always embedded within an avro file. It allows one to read the file without knowing the schema before-hand. The name [Avro](https://en.wikipedia.org/wiki/Avro) is borrowed from a defunct British aircraft manufacturer.

A producer that writes records to an avro file must specify the schema that describes the structure of each record. That avro schema is expressed in JSON. However, a higher abstraction, the Avro IDL language, lets developers specify schemas in a form readable for people. We'll explore IDL in a later lesson. For now, let's see an example using the JSON schema.

## Example

We will create an avro file consisting of records representing a car. But first, let's define the schema for each record. Avro schemas are defined using JSON. They can contain primitive (boolean, int, long) or complex types ( map, array, enum). The JSON schema for our car record looks like this:

```json
{
  "namespace": "datajek.io.avro",
  "type": "record",
  "name": "Car",
  "fields": [
    {
      "name": "make",
      "type": "string"
    },
    {
      "name": "model",
      "type": [
        "string",
        "null"
      ]
    },
    {
      "name": "year",
      "type": [
        "int",
        "null"
      ]
    },
    {
      "name": "horsepower",
      "type": [
        "int",
        "null"
      ]
    }
  ]
}
```

Using the above schema, we can create an .avro file consisting of car records. Each record is an instance of the interface `GenericRecord`; the fields are written as key/value pairs. Using the `GenericRecord` interface to represent records we lose type-safety and also need casts when accessing the record fields on a read-back from the avro file. Alternatively, we can generate Java classes representing the record from the schema using avro tools. We can work with the generated code, instead of generic records.


# Avro

Avro is a data serialization system that allows for remote procedure calls and the fast and compact serialization of data. The defining feature of Avro is a schema always embedded within an avro file. It allows one to read the file without knowing the schema before-hand. The name [Avro](https://en.wikipedia.org/wiki/Avro) is borrowed from a defunct British aircraft manufacturer.

A producer that writes records to an avro file must specify the schema that describes the structure of each record. That avro schema is expressed in JSON. However, a higher abstraction, the Avro IDL language, lets developers specify schemas in a form readable for people. We'll explore IDL in a later lesson. For now, let's see an example using the JSON schema.

# Example

We will create an avro file consisting of records representing a car. But first, let's define the schema for each record. Avro schemas are defined using JSON. They can contain primitive (boolean, int, long) or complex types ( map, array, enum). The JSON schema for our car record looks like this:

```json
{
  "namespace": "datajek.io.avro",
  "type": "record",
  "name": "Car",
  "fields": [
    {
      "name": "make",
      "type": "string"
    },
    {
      "name": "model",
      "type": [
        "string",
        "null"
      ]
    },
    {
      "name": "year",
      "type": [
        "int",
        "null"
      ]
    },
    {
      "name": "horsepower",
      "type": [
        "int",
        "null"
      ]
    }
  ]
}
```

Using the above schema, we can create an .avro file consisting of car records. Each record is an instance of the interface `GenericRecord`; the fields are written as key/value pairs. Using the `GenericRecord` interface to represent records we lose type-safety and also need casts when accessing the record fields on a read-back from the avro file. Alternatively, we can generate Java classes representing the record from the schema using avro tools. We can work with the generated code, instead of generic records.


This lesson explains the Avro serialization system.

Avro: Intro

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Avro: Intro

Avro