Learn about big data and the Hadoop ecosystem, and explore streaming technologies. Gain the essential context to kick-start your big data journey confidently. Perfect for beginners.

Intro to Big Data, Hadoop, and Streaming-01.jpg

hadoop1.tar.gz

HDFS_NAMENODE_USER

HDFS_SECONDARYNAMENODE_USER

HDFS_DATANODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

Hadoop

Docker Compose

This course will help introduce new learners  to the field of big data, Hadoop ecosystem, and streaming. Perfect for beginners, this concise course will give you the context you need to start your big data journey on the right foot.

The No Nonsense Introduction To Big Data, Hadoop and Streaming

In this lesson, we will look at an example of how we could use Kafka with a simple consumer and producer. We will also demonstrate how we can process streaming input with Kafka. For this purpose, we will use two Python scripts, one to feed Kafka, `producer.py` and one to consume from it, `consumer.py`.

As we explained before, Kafka supports many connection types so, instead of Python scripts, we could have any project like Apache Storm or a MySQL database.


For this demonstration, the producer will generate random numbers and the consumer throws an alert for the odd ones.

# Example

Click on the "Run" button at the bottom of the widget. This will bring up a cluster of Kafka and Zookeeper, which is a coordinator for distributed systems like Kafka.

Now, move to the `usercode` directory where all the files are stored. We have already pre-installed the Kafka dependency for you.

```
cd ../
```
To execute the program, we first need to run the following command: `pip3 install kafka-python`


Learn more about streaming with a simple example.

Before We Begin

Setting The Stage

The Hadoop Ecosystem

Streaming

Apache Spark

Conclusion

A Streaming Example