Request Processing

The job of a broker is to process requests received from the controller, partition replicas, and client requests that were sent to the partition leader. A request consists of the following elements:

• Request Type: Also known as the API key.
• Request Version: Allows brokers to handle clients of different versions.
• Correlation ID: Unique identifier for the request and also appears as part of the response in the error logs.
• Client ID: Identifier for the client application sending the request.

Kafka uses a binary protocol over TCP to define how brokers and clients communicate amongst themselves. The protocol is defined in detail here.

A broker has an acceptor thread listening on ports for incoming connections. Once a connection is established, the acceptor thread hands off the request for processing to the processor thread. The number of processor threads is configurable. The job of a processor thread is to take requests from client connections, place them in a request queue, pick responses from the response queue, and send them back to the client. The IO threads do the heavy lifting of actually fetching the data from storage. They read from the request queue and place responses in the response queue.