Description
The ZK producer, when used with the async producer option does the following
1. Create a pool of async producers, one each for a broker registered under /broker/ids
2. On each send request, apply the Partitioner, to decide the broker and partition to send the data
3. Use the Async producer's send API to enqueue that data into the async producer's queue
4. When the data is dequeued by the ProducerSendThread, use the underlying sync producer to send it to the broker
The load balancing decision is taken in step 2, before entering the queue. This leaves a window of error, equal to the queue length, when a broker can go down. When this happens, potentially, a queue worth of data can fail to reach a broker, and will be dropped by the EventHandler.
To correct this, the Producer, with the async option, needs to be refactored to allow only a single queue to hold all requests. And the application of the Partitioner should be moved to the end of the queue, in the EventHandler.
Attachments
Issue Links
- is part of
-
KAFKA-253 Refactor the async producer to have only one queue instead of one queue per broker in a Kafka cluster
-
- Resolved
-
