Request Processing
This lesson explains how requests are processed by Kafka brokers.
We'll cover the following...
The job of a broker is to process requests received from the controller, partition replicas, and client requests that were sent to the partition leader. A request consists of the following elements:
- Request Type: Also known as the API key.
- Request Version: Allows brokers to handle clients of different versions.
- Correlation ID: Unique identifier for the request and also appears as part of the response in the error logs.
- Client ID: Identifier for the client application sending the request.
Kafka uses a binary protocol over TCP to define how brokers and clients communicate amongst themselves. The protocol is defined in detail here.
A broker has an acceptor thread listening on ports for incoming connections. Once a connection is established, the acceptor thread hands off the request for processing to the processor thread. The number of processor threads is configurable. The job of a processor thread is to take requests from client connections, place them in a request queue, pick responses from the response queue, and send them back to the client. The IO threads do the ...