[KAFKA-13383] Consumer Group partition management across clusters (Active-Active)

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.1, 3.0.0
• Component/s: consumer, core, mirrormaker
• Assignee: Unassigned
• Reporter: NEERAJ VAIDYA
• Created: 18/Oct/21 23:29
• Updated: 18/Oct/21 23:29

Description

When using MM2 (Mirror Maker 2) in an Active-Active setup, we would like consumers to consume from the same topic name in both datacentres, rather than consume from 1) The topic without the site prefix 2) and the topic with the site prefix.

MM2 replicates a topic to a remote site by prefixing the site name.

Confluent replicator prevents this topic name prefixing by using Provenance headers in the message.

However, if we start 2 consumers (1 on each site) and both of them consume from the topic with the same name, the partitions are not balanced across sites.

So, the consumer on both sites process all messages and hence duplication of processing occurs. This might not be a big deal for consumers in some business domains, but in the Telecom domain, where duplicates are not tolerated, it becomes an issue.

It would be great if the partition allocation can be done across clusters.

So that when a consumer is started on 1 site, it always also takes into consideration if there are any other consumers for the same topic on the other site.