Example: Topics & Partitions
In this lesson, we'll study how our example is divided into topics and partitions.
Technical parameters of the partitions and topics #
The topic order contains the order records. Docker Compose configures the Kafka Docker container based on the environment variable KAFKA_CREATE_TOPICS in the file docker-compose.yml in such a way as to create the topic order.
The topic order is divided into five partitions, as a greater number of
partitions allows for more concurrency. In the example scenario, it is
not important to have a high degree of concurrency. More partitions
require more file handles on the server and more memory on the client.
When a Kafka node fails, it might be necessary to choose a new leader
for each partition. This also takes longer when more
partitions exist.
This argues for a lower number of partitions as used in the example in order to save resources. The number of partitions in a topic can still be changed after creating a topic.
However, in that case, the mapping of records to partitions will change. This can cause problems because then ...