[KAFKA-16254] Allow MM2 to fully disable offset sync feature

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0, 3.6.0, 3.7.0
• Fix Version/s: 3.9.0
• Component/s: mirrormaker
• Labels: need-kip
• Assignee: Omnia Ibrahim
• Reporter: Omnia Ibrahim
• Created: 14/Feb/24 16:32
• Updated: 09/Jul/24 10:42
• Resolved: 09/Jul/24 10:38

Description

Background:
At the moment syncing offsets feature in MM2 is broken to 2 parts

1. One is in `MirrorSourceTask` where we store the new recored's offset on target cluster to offset_syncs internal topic after mirroring the record. 
   Before KAFKA-14610 in 3.5 MM2 used to just queue the offsets and publish them later but since 3.5 this behaviour changed we now publish any offset syncs that we've queued up, but have not yet been able to publish when `MirrorSourceTask.commit` get invoked. This introduced an over head to commit process.
2. The second part is in checkpoints source task where we use the new record offsets from offset_syncs and update checkpoints and __consumer_offsets topics.

Problem:
For customers who only use MM2 for mirroring data and not interested in syncing offsets feature they now can disable the second part of this feature which is by disabling emit.checkpoints.enabled and/or sync.group.offsets.enabled to disable emitting __consumer_offsets topic but nothing disabling 1st part of the feature. 

The problem get worse if they disabled MM2 from creating offset syncs internal topic as
1. this will increase throughput as MM2 will try to force trying to update the offset with every mirrored batch which impacting the performance of our MM2.
2. Get too many error logs because they don't create the sync offset topic as they don't use the feature.

Possible solution:
Allow customers to fully disable the feature if they don't really need it similar to how we fully can disable other MM2 features like heartbeat feature by adding a new config.

Issue Links

• links to: GitHub Pull Request #15999
• mentioned in: Page Loading...