[KAFKA-9965] Uneven distribution with RoundRobinPartitioner in AK 2.4+

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0, 2.5.0, 2.4.1
• Component/s: producer
• Assignee: Unassigned
• Reporter: Michael Bingham
• Created: 06/May/20 23:44
• Updated: 02/Nov/23 08:25

Description

RoundRobinPartitioner states that it will provide equal distribution of records across partitions. However with the enhancements made in KIP-480, it may not. In some cases, when a new batch is started, the partitioner may be called a second time for the same record:

https://github.com/apache/kafka/blob/2.4/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L909

https://github.com/apache/kafka/blob/2.4/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L934

Each time the partitioner is called, it increments a counter in RoundRobinPartitioner, so this can result in unequal distribution.

Easiest fix might be to decrement the counter in RoundRobinPartitioner#onNewBatch.

 

Issue Links

• duplicates: KAFKA-13303 RoundRobinPartitioner broken by KIP-480
• links to: GitHub Pull Request #8633
• links to: GitHub Pull Request #8690
• links to: GitHub Pull Request #11326
• links to: GitHub Pull Request #12462