Detailed Design of Kafka

Kafka is a messaging system that includes a sender that publishes messages (data) not specifically to a receiver but assigns a type to them and a receiver that subscribes to a certain type of messages. Kafka also has a broker that facilitates both the sender and receiver publishing and subscribing to messages.

Kafka’s Architecture

Kafka is composed of three main components, which are described as follows.

Producers

A producer is responsible for creating new messages. They contain the following components:

• Producer record: A record is kept for each message created in the producer record. It is composed of the topic that’s supposed to get the message, a key or partition, and a value.

• Serializer: The first thing a producer does on a message is that it converts the key and value of a message into byte arrays (this process is sometimes called the marshaling of data).

• Partitioner: After being serialized, the message goes to the partitioner, which returns the partitions of a specific topic to which the message should be assigned to. Following are a few ways a partitioner can operate on messages:

  • If we specify a key in the producer record, the partitioner uses a hash function on the message key and maps it to ...