[KAFKA-15372] MM2 rolling restart can drop configuration changes silently

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.7.0, 3.6.2
• Component/s: mirrormaker
• Assignee: Greg Harris
• Reporter: Daniel Urban
• Created: 17/Aug/23 15:09
• Updated: 14/Feb/24 19:42
• Resolved: 12/Dec/23 21:58

Description

When MM2 is restarted, it tries to update the Connector configuration in all flows. This is a one-time trial, and fails if the Connect worker is not the leader of the group.

In a distributed setup and with a rolling restart, it is possible that for a specific flow, the Connect worker of the just restarted MM2 instance is not the leader, meaning that Connector configurations can get dropped.

For example, assuming 2 MM2 instances, and one flow A->B:

1. MM2 instance 1 is restarted, the worker inside MM2 instance 2 becomes the leader of A->B Connect group.
2. MM2 instance 1 tries to update the Connector configurations, but fails (instance 2 has the leader, not instance 1)
3. MM2 instance 2 is restarted, leadership moves to worker in MM2 instance 1
4. MM2 instance 2 tries to update the Connector configurations, but fails

At this point, the configuration changes before the restart are never applied. Many times, this can also happen silently, without any indication.

Issue Links

• fixes: KAFKA-10719 MirrorMaker2 fails to update its runtime configuration
• relates to: KAFKA-10586 Full support for distributed mode in dedicated MirrorMaker 2.0 clusters
• links to: GitHub Pull Request #14245
• links to: GitHub Pull Request #14293