Partition Allocation
Explore Kafka's partition allocation process, including leader and replica placement across brokers. Learn about rack awareness to improve fault tolerance and how partition directories are chosen to balance load on brokers.
We'll cover the following...
Kafka follows an algorithm when assigning partitions of a topic to brokers. Kafka attempts to evenly divide the partitions among the brokers as much as possible. As an example, consider a topic with 5 partitions and a replication factor of 3, making for a total of 15 partition replicas to be allocated amongst 5 brokers. The general algorithm will proceed as follows:
-
There should be 3 partition replicas per broker.
-
Initially, we start with determining the locations for the partition replica leaders. In our example, the replica leader for partition 0 is placed on broker 1, the replica leader for partition 1 is placed on broker 2, the replica leader for partition 2 onto broker 3, and so on and so forth. A pictorial representation is shown below: ...
| Broker 1 | Broker 2 | Broker 3 | Broker 4 | Broker 5 |
|---|---|---|---|---|
| Leader Partition#0 | Leader Partition#1 | Leader |