Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

# Terminology  

A reader will come across lots of confusing, technical jargon when starting to read about data storage and processing. We'll briefly discuss the various commonly used terms in the industry throughout the course so that the reader can visualize where Big Data fits in the broader scheme of technologies. 

# Data

Let's start with data. Data (plural of datum) are distinct pieces of information. Data can exist in several different forms: numbers, text, bytes, Instagram pictures, or YouTube videos. These represent various types of data that can be stored and transmitted electronically. Note that data are usually interpreted in a context. For example, an ASCII text line representing prose in the English language can't be interpreted as a .jpeg picture and vice versa.

There are broadly three categories of data:

1. __Structured data__: has some pre-defined organizational property making it easily searchable and analyzable. The data is backed by a model that dictates the size of each field: its type, length, and restrictions on what values it can take on. Data stored in SQL databases is structured. Structured data is usually formatted in a universally understandable and identifiable manner. In most instances, a schema formally specifies structured data. When working in any variant of SQL, you are almost always dealing with structured data.

2. __Unstructured data__: is characterized by a lack of organization and data model that describes the structure of a single record or attributes of any individual fields within the record. Videos, audio, blogs, log files, and social-media posts are all examples of unstructured data. It is data without any conceptual definition or type.


3. __Semi-structured data__: is a cross between structured and unstructured data. Even though there's no explicit data model or structure definition, one may be implied. Semi-structured data contains semantic tags but does not conform to the structure associated with typical relational databases. Examples include JSON and XML data. It sits in between the spectrum of structured and unstructured data. Another example is the metadata related to videos and audios files, which themselves fall under unstructured data. The metadata such as creator, last accessed, permissions, and more are semi-structured data.

Semi-structured data contains both structured, and unstructured parts. X-rays and other large images consist of mostly unstructured data that comprises of millions of pixels. It is impossible to search and query these X-ray images  like one can search and query a large relational database. Even though the files themselves consist of only pixels, small sections known as metadata within each file consists of details such as author, creation timestamp, and last accessed. This metadata allows for some analysis of unstructured data.

This brings us to the question: if all unstructured data has associated metadata then what is the difference between semi-structured and unstructured data? There's hardly any truly unstructured data with no organization or metadata. In fact, the distinction between semi-structured and unstructured data is a disputed grey area. However, both are far from the structured and rigorously organized data living in relational databases. Often what is referred to as unstructured data (videos, images, documents, social media postings, files) is really semi-structured data. But, for simplicity data are usually divided and described as structured or unstructured.


Structured data is usually handled by a database management system. We'll explore that next.

# Database Management System (DBMS)

Usually, when referring to databases such as MySQL and PostgreSQL, we are talking about a system, called the database management system. A DBMS is a  software that allows the user to create, maintain, and delete multiple individual databases.

It provides peripheral services and interfaces for the end-user to interact with the databases.

# Database

A database is an organized and structured collection of data, usually stored and retrieved electronically. A database usually lives in a database management system. The structure and organization helps achieve efficient data retrieval.

There are broadly two types of databases:

This lesson describes the differences between the various categories of data.

Untitled Masterpiece

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Untitled Masterpiece

Terminology

Data