Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-13383

Consumer Group partition management across clusters (Active-Active)

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.8.1, 3.0.0
    • None
    • consumer, core, mirrormaker
    • None

    Description

      When using MM2 (Mirror Maker 2) in an Active-Active setup, we would like consumers to consume from the same topic name in both datacentres, rather than consume from 1) The topic without the site prefix 2) and the topic with the site prefix.

      MM2 replicates a topic to a remote site by prefixing the site name.

      Confluent replicator prevents this topic name prefixing by using Provenance headers in the message.

      However, if we start 2 consumers (1 on each site) and both of them consume from the topic with the same name, the partitions are not balanced across sites.

      So, the consumer on both sites process all messages and hence duplication of processing occurs. This might not be a big deal for consumers in some business domains, but in the Telecom domain, where duplicates are not tolerated, it becomes an issue.

      It would be great if the partition allocation can be done across clusters.

      So that when a consumer is started on 1 site, it always also takes into consideration if there are any other consumers for the same topic on the other site.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            neeraj.vaidya NEERAJ VAIDYA

            Dates

              Created:
              Updated:

              Slack

                Issue deployment