[KAFKA-10586] Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: 3.5.0
• Component/s: mirrormaker
• Labels: cloudera
• Assignee: Chris Egerton
• Reporter: Daniel Urban
• Created: 08/Oct/20 14:22
• Updated: 04/Mar/24 19:25
• Resolved: 14/Feb/23 21:37

Description

KIP-382 introduced MirrorMaker 2.0. Because of scoping issues, the dedicated MirrorMaker 2.0 cluster does not utilize the Connect REST API. This means that with specific workloads, the dedicated MM2 cluster can become unable to react to dynamic topic and group filter changes.

(This occurs when after a rebalance operation, the leader node has no MirrorSourceConnectorTasks. Because of this, the MirrorSourceConnector is stopped on the leader, meaning it cannot detect config changes by itself. Followers still running the connector can detect config changes, but they cannot query the leader for config updates.)

Besides the REST support, config provider references should be evaluated lazily in connector configurations.

Issue Links

• fixes: KAFKA-9981 Running a dedicated mm2 cluster with more than one nodes,When the configuration is updated the task is not aware and will lose the update operation.
• fixes: KAFKA-14898 [ MirrorMaker ] sync.topic.configs.enabled not working as expected
• fixes: KAFKA-10582 Mirror Maker 2 not replicating new topics until restart
• is related to: KAFKA-10857 Mirror Maker 2 - replication not working when deploying multiple instances
• is related to: KAFKA-15372 MM2 rolling restart can drop configuration changes silently

Sub-Tasks

• MM2 should delay resolving config provider references: Resolved