Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

## Filesystem


A file system is a mechanism of storing, organizing and retrieving data on a storage medium. The fundamental unit of a file system is a _file_, logically related named piece of data. The structure and rules used to manage files and their names comprises a file system.

## Types of file systems
There are different types of file systems each exhibiting different characteristics with respect to reliability, security, and performance. Examples are:

+ Disk File Systems
+ Tape File Systems
+ Network File Systems
+ Special Purpose File Systems
  

## OS and file systems

Operating systems generally support more than one file system. For instance, Apple's MacOS uses __APFS__(Apple File System), which replaced an earlier file system called __HFS+__. Windows supports __FAT__, its variants and NTFS file systems, and Linux supports __EXT__ family of file systems.

## Distributed file systems
In the context of Hadoop, we use distributed file systems (DFS). A DFS uses the network to send/receive data and creates the illusion of a local file system for clients. Access to files is provided from multiple hosts. Files may be stored on a central server or across multiple servers. There are several distributed file systems in operation today, including [Lustre](https://en.wikipedia.org/wiki/Lustre_(file_system)), [Google File System (GFS)](https://en.wikipedia.org/wiki/Google_File_System), and [Andrew File System](https://en.wikipedia.org/wiki/Andrew_File_System).

Next we'll examine the building block of a filesystem.

## Disk block

A disk block is the smallest unit writable by a disk or file system. Everything a file system does is composed of operations executed on disk blocks. However, file systems don't write individual disk blocks. Rather they read/write a couple of blocks together for efficiency. This abstraction over the physical disk blocks is called the filesystem block. A file system block is always the same size as or larger (in integer multiples) than the disk block size.

## File metadata
The name of a file is metadata because it is information about the file not in the stream of bytes making up the file. There are several other pieces of metadata about a file like the owner, security access controls, date of last modification, creation time, and size. Generally the file system stores file metadata in an i-node or index-node. Examples of information stored in i-nodes are the last access time of the file, type, creator, version number, and reference to the directory that contains the file.

In Hadoop, the file's metadata is kept separate from the file data, as we'll later learn.


## i-Node

In a traditional filesystems (like Unix or its variants), the i-node (index node) is a data structure in the filesystem that stores metadata information and information about data's location on the storage media. An i-node refers to the contents of the file by tracking a list of blocks on the disk that belong to this file. A file appears as a continuous stream of bytes at higher levels; the blocks that contain the file data may not be contiguous on disk. An i-node contains the information the file system uses to map from a logical position in a file (for example, byte offset 11,239) to a physical position on disk.

We'll see a parallel of the i-node in HDFS too.

# Filesystem


A file system is a mechanism of storing, organizing and retrieving data on a storage medium. The fundamental unit of a file system is a _file_, logically related named piece of data. The structure and rules used to manage files and their names comprises a file system.

# Types of file systems
There are different types of file systems each exhibiting different characteristics with respect to reliability, security, and performance. Examples are:

+ Disk File Systems
+ Tape File Systems
+ Network File Systems
+ Special Purpose File Systems
  

# OS and file systems

Operating systems generally support more than one file system. For instance, Apple's MacOS uses __APFS__(Apple File System), which replaced an earlier file system called __HFS+__. Windows supports __FAT__, its variants and NTFS file systems, and Linux supports __EXT__ family of file systems.

# Distributed file systems
In the context of Hadoop, we use distributed file systems (DFS). A DFS uses the network to send/receive data and creates the illusion of a local file system for clients. Access to files is provided from multiple hosts. Files may be stored on a central server or across multiple servers. There are several distributed file systems in operation today, including [Lustre](https://en.wikipedia.org/wiki/Lustre_(file_system)), [Google File System (GFS)](https://en.wikipedia.org/wiki/Google_File_System), and [Andrew File System](https://en.wikipedia.org/wiki/Andrew_File_System).

Next we'll examine the building block of a filesystem.

# Disk block

A disk block is the smallest unit writable by a disk or file system. Everything a file system does is composed of operations executed on disk blocks. However, file systems don't write individual disk blocks. Rather they read/write a couple of blocks together for efficiency. This abstraction over the physical disk blocks is called the filesystem block. A file system block is always the same size as or larger (in integer multiples) than the disk block size.

# File metadata
The name of a file is metadata because it is information about the file not in the stream of bytes making up the file. There are several other pieces of metadata about a file like the owner, security access controls, date of last modification, creation time, and size. Generally the file system stores file metadata in an i-node or index-node. Examples of information stored in i-nodes are the last access time of the file, type, creator, version number, and reference to the directory that contains the file.

In Hadoop, the file's metadata is kept separate from the file data, as we'll later learn.


# i-Node

In a traditional filesystems (like Unix or its variants), the i-node (index node) is a data structure in the filesystem that stores metadata information and information about data's location on the storage media. An i-node refers to the contents of the file by tracking a list of blocks on the disk that belong to this file. A file appears as a continuous stream of bytes at higher levels; the blocks that contain the file data may not be contiguous on disk. An i-node contains the information the file system uses to map from a logical position in a file (for example, byte offset 11,239) to a physical position on disk.

We'll see a parallel of the i-node in HDFS too.

This lesson discusses the filesystem and its related entities.

Filesystem

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Filesystem

Filesystem

Types of file systems

OS and file systems