Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14757

Kafka Cooperative Sticky Assignor results in significant duplicate consumption

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 3.1.1
    • None
    • consumer
    • None
    • AWS MSK (broker) and Spring Kafka (2.8.7) for use in Spring Boot consumers.

    Description

      Details may be found within the linked document:

      [Kafka Cooperative Sticky Assignor Issue : Duplicate Consumption | https://docs.google.com/document/d/1E7qAwGOpF8jo_YhF4NwUx9CXxUGJmT8OhHEqIg7-GfI/edit?usp=sharing]

      In a nutshell, we noticed that the Cooperative Sticky Assignor resulted in significant duplicate message consumption. During last year's F1 Grand Prix events and World Cup soccer events, our company's Kafka-based platform received live-traffic. This live traffic, coupled with autoscaled consumers resulted in as much as 70% duplicate message consumption at the Kafka consumers. 

      In December 2022, we ran a synthetic load test to confirm that duplicate message consumption occurs during consumer scale out/in and Kafka partition rebalancing when using the Cooperative Sticky Assignor. This issue does not occur when using the Range Assignor.

       

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            sanand Siddharth Anand

            Dates

              Created:
              Updated:

              Slack

                Issue deployment