Kafka Controller

This lesson examines how Kafka brokers become members of a cluster and the role of a controller within a cluster.

We'll cover the following...

If you are unfamiliar with Zookeeper, we would suggest you read the Zookeeper chapter in the appendix first and then come back to this lesson.

Broker membership

So far, we have largely skipped the internal working details of a Kafka cluster and its interactions with consumers and producers. In this chapter, we’ll take a closer look at how the different Kafka components work. We’ll start with the Kafka cluster that consists of several brokers working together. Brokers maintain their membership in a cluster via a unique ID that is set either in the configuration file or automatically generated. Each broker creates an ephemeral node in Zookeeper with its ID under the Zookeeper path /brokers/id. Various Kafka components receive notifications when brokers join or leave the cluster by keeping a watch on the path /brokers/id where brokers create ...