[KAFKA-14757] Kafka Cooperative Sticky Assignor results in significant duplicate consumption

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: consumer
• Environment: AWS MSK (broker) and Spring Kafka (2.8.7) for use in Spring Boot consumers.
• Assignee: Unassigned
• Reporter: Siddharth Anand
• Created: 24/Feb/23 05:50
• Updated: 27/Mar/23 21:44

Description

Details may be found within the linked document:

[Kafka Cooperative Sticky Assignor Issue : Duplicate Consumption | https://docs.google.com/document/d/1E7qAwGOpF8jo_YhF4NwUx9CXxUGJmT8OhHEqIg7-GfI/edit?usp=sharing]

In a nutshell, we noticed that the Cooperative Sticky Assignor resulted in significant duplicate message consumption. During last year's F1 Grand Prix events and World Cup soccer events, our company's Kafka-based platform received live-traffic. This live traffic, coupled with autoscaled consumers resulted in as much as 70% duplicate message consumption at the Kafka consumers. 

In December 2022, we ran a synthetic load test to confirm that duplicate message consumption occurs during consumer scale out/in and Kafka partition rebalancing when using the Cooperative Sticky Assignor. This issue does not occur when using the Range Assignor.

 

Issue Links

• relates to: KAFKA-14224 Consumer should auto-commit prior to cooperative partition revocation