A Streaming Example

In this lesson, we will look at an example of how we could use Kafka with a simple consumer and producer. We will also demonstrate how we can process streaming input with Kafka. For this purpose, we will use two Python scripts, one to feed Kafka, producer.py and one to consume from it, consumer.py.

As we explained before, Kafka supports many connection types so, instead of Python scripts, we could have any project like Apache Storm or a MySQL database.

For this demonstration, the producer will generate random numbers and the consumer throws an alert for the odd ones.

Example

Click on the “Run” button at the bottom of the widget. This will bring up a cluster of Kafka and Zookeeper, which is a coordinator for distributed systems like Kafka.

Now, move to the usercode directory where all the files are stored. We have already pre-installed the Kafka dependency for you.

cd ../

To execute the program, we first need to run the following command: pip3 install kafka-python