Partitions

When sending messages to Kafka topics, we can choose whether or not to specify a key. If we don’t specify a key (i.e. it is set to null), Kafka’s default mechanism takes over to assign a partition. The default partitioner directs the message to one of the available partitions in a round-robin fashion.

If the key is specified but the partitioner class isn’t, then the default partitioner is employed. It will generate a hash of the key using its own algorithm so that the hash never changes on future Java upgrades. Messages with the same key will always have the same hash and be delivered to the same partition. The astute reader may wonder what happens if a partition is down and can’t be written to at the time the message is sent. Usually, partitions have replicas that can receive a message for processing when another replica is rendered out of service. However, adding more partitions to an existing topic doesn’t guarantee that new messages with a given key will land in the same partition as before. Generally, new partitions shouldn’t be added to a topic if partitioning by keys is important.