[KAFKA-14663] High throughput topics can starve low-throughput MM2 offset syncs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.1.0, 3.0.0, 3.3.0, 3.4.0, 3.5.0
• Component/s: mirrormaker
• Assignee: Greg Harris
• Reporter: Greg Harris
• Created: 31/Jan/23 01:08
• Updated: 27/Nov/23 19:10
• Resolved: 06/Feb/23 18:07

Description

In MM2, a semaphore is used to throttle the number of offset syncs written to the offset-syncs topic. If too many offset writes are requested (for example, from high-throughput topics) then some are silently dropped and never retried. This is acceptable for a single topic-partition, where a later record may re-trigger the offset-sync and write the sync successfully.

However, if there is a large variance between throughput in the topics emitted by an MM2 instance, it is possible for high-throughput topics to trigger many offset syncs, and cause the offset-syncs for a co-located low-throughput topic to be unfairly dropped.

This can cause the downstream offsets for the starved topic to lag behind significantly, or be prevented completely.

Instead, we should have some sort of fairness mechanism where low-thoughput topics are given similar priority to high-throughput topics in producing offset syncs, and cause excess sync messages from high-throughput topics to be dropped instead.

Issue Links

• Discovered while testing: KAFKA-12468 Initial offsets are copied from source to target cluster
• Discovered while testing: KAFKA-13562 Mirror Maker 2 Negative Offsets
• duplicates: KAFKA-14610 Publish Mirror Maker 2 offset syncs in task commit method
• relates to: KAFKA-15906 Emit offset syncs more often than offset.lag.max for low-throughput/finite partitions