Learn about big data and the Hadoop ecosystem, and explore streaming technologies. Gain the essential context to kick-start your big data journey confidently. Perfect for beginners.

Intro to Big Data, Hadoop, and Streaming-01.jpg

hadoop1.tar.gz

HDFS_NAMENODE_USER

HDFS_SECONDARYNAMENODE_USER

HDFS_DATANODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

Hadoop

Docker Compose

This course will help introduce new learners  to the field of big data, Hadoop ecosystem, and streaming. Perfect for beginners, this concise course will give you the context you need to start your big data journey on the right foot.

The No Nonsense Introduction To Big Data, Hadoop and Streaming

## Apache Kafka

Apache Kafka is one of the most famous streaming platforms. It was created by LinkedIn initially and, after a few years, was donated to the Apache Foundation.

Kafka is a distributed streaming platform. We can use it as a real-time messaging system with a high fault tolerance capability. In other words, it means that our messages will be delivered fast, and if it fails, we can figure it out quickly. A message can be anything from a string, to a serialized object, to a blob.

Kafka is also very famous for its scalability powers. Despite its numerous capabilities, Kafka is widely used to connect heterogeneous applications in a many-to-many manner.


For example, when the system receives a new stock price, it shows in the UI, updates the database with the new value, and stores the event of a change to HDFS or any data lake we use. We can do this by having multiple consumers, also referred to as connectors, read the message and act upon it.

Kafka has various subcomponents, like Kafka Streams and KafkaSQL, each of which has its own specialization.


## Apache Flume

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of data. Usually, this data refers to log files. For example, imagine running a web app using multiple servers, and we want to get the logs in one place. This would be a perfect use case for Apache Flume.

It has a simple and flexible architecture based on streaming data flows. Flume is robust and fault-tolerant, with tunable reliability mechanisms, and many failovers and recovery mechanisms.

## Spark Streaming
It is part of the core Spark API and fulfills the philosophy of being the “one and only tool for your big data processing.” Spark streaming provides streaming capabilities in a high-throughput, fault-tolerant way.

It accepts input from many different source types like Kafka or HDFS and pushes the data to a persistence layer in a streaming way.


## Apache Storm

Apache Storm was initially created on Twitter to perform real-time message processing and might look similar to Kafka. However, unlike Kafka, it focuses on computation rather than on delivery.

Kafka is mostly a queue, whereas Storm can do multiple types of computations over data.

## Alternatives

These tools have a steep learning curve. Therefore, they're not the best fit for a one-off gig or small projects.

Here are some tools that require less reading and can achieve results of a similar nature:
- The PUB-SUB functionality of the Redis database.
- RabbitMQ.
- WebSockets, to trigger some processing as a result of a new data event.



# Apache Kafka

Apache Kafka is one of the most famous streaming platforms. It was created by LinkedIn initially and, after a few years, was donated to the Apache Foundation.

Kafka is a distributed streaming platform. We can use it as a real-time messaging system with a high fault tolerance capability. In other words, it means that our messages will be delivered fast, and if it fails, we can figure it out quickly. A message can be anything from a string, to a serialized object, to a blob.

Kafka is also very famous for its scalability powers. Despite its numerous capabilities, Kafka is widely used to connect heterogeneous applications in a many-to-many manner.


For example, when the system receives a new stock price, it shows in the UI, updates the database with the new value, and stores the event of a change to HDFS or any data lake we use. We can do this by having multiple consumers, also referred to as connectors, read the message and act upon it.

Kafka has various subcomponents, like Kafka Streams and KafkaSQL, each of which has its own specialization.


# Apache Flume

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of data. Usually, this data refers to log files. For example, imagine running a web app using multiple servers, and we want to get the logs in one place. This would be a perfect use case for Apache Flume.

It has a simple and flexible architecture based on streaming data flows. Flume is robust and fault-tolerant, with tunable reliability mechanisms, and many failovers and recovery mechanisms.

# Spark Streaming
It is part of the core Spark API and fulfills the philosophy of being the “one and only tool for your big data processing.” Spark streaming provides streaming capabilities in a high-throughput, fault-tolerant way.

It accepts input from many different source types like Kafka or HDFS and pushes the data to a persistence layer in a streaming way.


# Apache Storm

Apache Storm was initially created on Twitter to perform real-time message processing and might look similar to Kafka. However, unlike Kafka, it focuses on computation rather than on delivery.

Kafka is mostly a queue, whereas Storm can do multiple types of computations over data.

# Alternatives

These tools have a steep learning curve. Therefore, they're not the best fit for a one-off gig or small projects.

Here are some tools that require less reading and can achieve results of a similar nature:
- The PUB-SUB functionality of the Redis database.
- RabbitMQ.
- WebSockets, to trigger some processing as a result of a new data event.



Learn about some of the common real-time streaming platforms.

Before We Begin

Setting The Stage

The Hadoop Ecosystem

Streaming

Apache Spark

Conclusion

Real-Time Streaming Platforms

Apache Kafka