Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

## SerDe

In this lesson, we'll study why serialization is important and which serialization systems are used in the Big Data ecosystem. _SerDe_ is short for Serializer/Deserializer. Encoding an object as a byte stream is called serializing the object. Once an object is serialized, its encoding can be transmitted from one running virtual machine to another, and stored on disk or in a database for deserialization later. Serialization is often used in network programming. Objects that need to be transmitted through the network must be converted into bytes. Serialization can also store objects on disk.

## Need for SerDe

Why can't we use Java's own serialization capability when most of Hadoop daemons are written in Java? Most languages offer libraries for serialization like pickle in Python and marshal in Ruby. However, when we serialize an object using a particular technology, the serialized object cannot be deserialized using another technology. An object serialized by a Java program can't be read-in (deserialized) by a Python program.

Another way to solve the interoperability issues is to serialize using JSON or XML. But these formats is that they are verbose and slow to process. Additionally, it's not possible to distinguish between different data types when using these formats. For example, decimal can't be told apart from an integer. Binary and unicode strings do not easily fit into JSON or XML. In the case of JSON, the field name for each attribute of an object is repeated for every object adding bytes to the total size of the serialized file. Some of these problems are mitigated by newer serialization formats that are essentially a binary take on JSON, such as BJSON.

In short, all of the choices above come with their own set of shortcomings. Ideally, a serialization system should address the following concerns:

+ __Minimizing Data on Wire:__ Transferring data over the wire adds to latency. We want to reduce the time it takes to transfer data. One solution is making the network connection faster; the second is reducing the amount of data transferred. Various serialization systems store objects in a format that can be easily sent over a network with the minimal possible number of bytes.

+ __Portability:__ When a system sends one serialized object to another, the receiving system needs enough information to reconstruct the object from the received byte stream.

+ __Versioning:__ Data objects evolve over time. New fields or attributes may be added and old ones deleted. A serialization framework needs to provide ability for the users to work with changing schemas.

+ __Data Integrity__: Another goal of serialization is data typing and integrity. A field sent as a string should not be interpreted as an integer on the receiving end.

While within the Big Data ecosystem, data needs to be transferred across nodes and stored on hard disks. The format in which the data gets stored can have important consequences later, when the organization grows and data is produced and consumed by various teams. Usually, companies store data as CSV or JSON files when starting with Big Data. However, these two formats introduce the overhead of parsing text each time data is read. Secondly, the data doesn't have an accompanying, fully-defined schema such as in the case of a relational database. A column in CSV or a field in JSON represents some data type. It is interpreted as another in scripts and programs that consume the data. The schema is informally encoded in the program logic or exists in the heads of the engineers. This brittle setup breaks when the data producers add new fields or change the types of the columns. Suddenly, programs start to fail.

The solutions to several of these ills is to use the newer, shiner serialization systems that compactly store data with the schema. The schema is not repeated per record, it is with JSON. Some popular data serialization systems are listed below. We will explore a few of them in the upcoming lessons.

+ Avro
+ Parquet
+ Thrift
+ Protocol Buffers

# SerDe

In this lesson, we'll study why serialization is important and which serialization systems are used in the Big Data ecosystem. _SerDe_ is short for Serializer/Deserializer. Encoding an object as a byte stream is called serializing the object. Once an object is serialized, its encoding can be transmitted from one running virtual machine to another, and stored on disk or in a database for deserialization later. Serialization is often used in network programming. Objects that need to be transmitted through the network must be converted into bytes. Serialization can also store objects on disk.

# Need for SerDe

Why can't we use Java's own serialization capability when most of Hadoop daemons are written in Java? Most languages offer libraries for serialization like pickle in Python and marshal in Ruby. However, when we serialize an object using a particular technology, the serialized object cannot be deserialized using another technology. An object serialized by a Java program can't be read-in (deserialized) by a Python program.

Another way to solve the interoperability issues is to serialize using JSON or XML. But these formats is that they are verbose and slow to process. Additionally, it's not possible to distinguish between different data types when using these formats. For example, decimal can't be told apart from an integer. Binary and unicode strings do not easily fit into JSON or XML. In the case of JSON, the field name for each attribute of an object is repeated for every object adding bytes to the total size of the serialized file. Some of these problems are mitigated by newer serialization formats that are essentially a binary take on JSON, such as BJSON.

In short, all of the choices above come with their own set of shortcomings. Ideally, a serialization system should address the following concerns:

+ __Minimizing Data on Wire:__ Transferring data over the wire adds to latency. We want to reduce the time it takes to transfer data. One solution is making the network connection faster; the second is reducing the amount of data transferred. Various serialization systems store objects in a format that can be easily sent over a network with the minimal possible number of bytes.

+ __Portability:__ When a system sends one serialized object to another, the receiving system needs enough information to reconstruct the object from the received byte stream.

+ __Versioning:__ Data objects evolve over time. New fields or attributes may be added and old ones deleted. A serialization framework needs to provide ability for the users to work with changing schemas.

+ __Data Integrity__: Another goal of serialization is data typing and integrity. A field sent as a string should not be interpreted as an integer on the receiving end.

While within the Big Data ecosystem, data needs to be transferred across nodes and stored on hard disks. The format in which the data gets stored can have important consequences later, when the organization grows and data is produced and consumed by various teams. Usually, companies store data as CSV or JSON files when starting with Big Data. However, these two formats introduce the overhead of parsing text each time data is read. Secondly, the data doesn't have an accompanying, fully-defined schema such as in the case of a relational database. A column in CSV or a field in JSON represents some data type. It is interpreted as another in scripts and programs that consume the data. The schema is informally encoded in the program logic or exists in the heads of the engineers. This brittle setup breaks when the data producers add new fields or change the types of the columns. Suddenly, programs start to fail.

The solutions to several of these ills is to use the newer, shiner serialization systems that compactly store data with the schema. The schema is not repeated per record, it is with JSON. Some popular data serialization systems are listed below. We will explore a few of them in the upcoming lessons.

+ Avro
+ Parquet
+ Thrift
+ Protocol Buffers

This lesson explains the importance of serialization in the Hadoop landscape.

SerDe

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

SerDe

SerDe

Need for SerDe