Gain insights into Apache Kafka's role in scalable data pipelines. Explore its theory and practice interactive commands to build efficient and diverse data transmission solutions.

kafka.tar.gz

If you’re interested in Big Data, then Apache Kafka is a must-know tool. 

What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scalable data pipelines. Meet Apache Kafka - the ubiquitous tool to build pipelines for diverse use cases ranging from chronologically tracking user-activity on a website to implementing publish-subscribe feeds. 

This course introduces you to Kafka theory and provides you with a hands-on interactive browser-terminal to execute Kafka commands against a running Kafka broker.

Building Scalable Data Pipelines with Kafka

Kafka consumers are instances of the class `KafkaConsumer`, which is a parallel of the `KafkaProducer` class used to instantiate producers. The `KafkaConsumer` class requires three compulsory properties to be provided: the location of the servers `bootstrap.servers`, the key deserializer `key.deserializer`, and the value deserializer `value.deserializer`. The deserializers convert byte arrays into Java objects. 

When creating a `KafkaConsumer` we can also specify a consumer group id using the property `group.id`. We can create Kafka consumers that don't belong to any consumer group, but this practice is uncommon. When subscribing to topics, the consumer has the choice to subscribe to a single topic or use a regular expression to match multiple topics.

## Poll loop
The general pattern for Kafka consumers is to poll for new messages and process them in a perpetual loop, often referred to as the **poll loop**. Within the poll loop, the `poll()` method takes a timeout interval that the consumer blocks for when there's no data in the consumer buffer. If the interval value is set to `0`, the method returns immediately. It is essential that the consumer keeps polling the broker, as the "I am alive" heartbeats are sent as part of the poll method. 

In the newer versions of Kafka, the heartbeat can be configured to send messages in-between consumer application data polling requests. In the older versions of Kafka, where the `poll()` method solely sent out the heartbeats, a consumer may be presumed to be dead because of a long interval between consecutive polls. 

The long interval can be caused by several factors, including a prolonged time to process the received data from the previous poll call, a garbage collection pause in the JVM, or simply an OS running on hardware with limited capabilities having a hard time scheduling consumer applications for processing.

The first time `poll()` is invoked by a new consumer, the invocation is responsible for finding the `GroupCoordinator`, joining the consumer group, and receiving a partition assignment.

The following code widget demonstrates the code written for a Kafka consumer that polls for events from all topics matching the regular expression `datajek-*`. The poll loop appears highlighted in the widget.

Kafka consumers are instances of the class `KafkaConsumer`, which is a parallel of the `KafkaProducer` class used to instantiate producers. The `KafkaConsumer` class requires three compulsory properties to be provided: the location of the servers `bootstrap.servers`, the key deserializer `key.deserializer`, and the value deserializer `value.deserializer`. The deserializers convert byte arrays into Java objects. 

When creating a `KafkaConsumer` we can also specify a consumer group id using the property `group.id`. We can create Kafka consumers that don't belong to any consumer group, but this practice is uncommon. When subscribing to topics, the consumer has the choice to subscribe to a single topic or use a regular expression to match multiple topics.

# Poll loop
The general pattern for Kafka consumers is to poll for new messages and process them in a perpetual loop, often referred to as the **poll loop**. Within the poll loop, the `poll()` method takes a timeout interval that the consumer blocks for when there's no data in the consumer buffer. If the interval value is set to `0`, the method returns immediately. It is essential that the consumer keeps polling the broker, as the "I am alive" heartbeats are sent as part of the poll method. 

In the newer versions of Kafka, the heartbeat can be configured to send messages in-between consumer application data polling requests. In the older versions of Kafka, where the `poll()` method solely sent out the heartbeats, a consumer may be presumed to be dead because of a long interval between consecutive polls. 

The long interval can be caused by several factors, including a prolonged time to process the received data from the previous poll call, a garbage collection pause in the JVM, or simply an OS running on hardware with limited capabilities having a hard time scheduling consumer applications for processing.

The first time `poll()` is invoked by a new consumer, the invocation is responsible for finding the `GroupCoordinator`, joining the consumer group, and receiving a partition assignment.

The following code widget demonstrates the code written for a Kafka consumer that polls for events from all topics matching the regular expression `datajek-*`. The poll loop appears highlighted in the widget.

This lesson explains the Kafka consumer and its programmatic equivalent, the KafkaConsumer class..

Basics

Kafka Producer

Kafka Consumer

Kafka Internals

Conclusion

Appendix

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Kafka Consumer