[KAFKA-253] Refactor the async producer to have only one queue instead of one queue per broker in a Kafka cluster

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.8.0
• Assignee: Jun Rao
• Reporter: Neha Narkhede
• Created: 24/Jan/12 19:39
• Updated: 28/Mar/13 15:40
• Resolved: 15/May/12 01:00

Description

Today, the async producer is associated with a particular broker instance, just like the SyncProducer. The Producer maintains a producer pool of sync/async producers, one per broker. Since the producer pool creates one async producer per broker, we have multiple producer queues for one Producer instance.

With replication, a topic partition will be logical. This requires refactoring the AsyncProducer to be broker agnostic. As a side effect of this refactoring, we should also ensure that we have only one queue per Producer instance.

Issue Links

• blocks: KAFKA-239 Wire existing producer and consumer to use the new ZK data structure
• incorporates: KAFKA-233 The producer's load balancing logic can send requests to dead brokers, when using the async producer option