[KAFKA-16641] MM2 offset translation should interpolate between sparse OffsetSyncs

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: mirrormaker
• Assignee: Unassigned
• Reporter: Greg Harris
• Created: 29/Apr/24 17:39
• Updated: 23/May/24 11:38

Description

Right now, the OffsetSyncStore keeps a sparse offset store, with exponential spacing between syncs. This can leave large gaps in translation, where offsets are translated much more conservatively than necessary.

The dominant way to use MirrorMaker2 is in a "single writer" fashion, where the target topic is only written to by a single mirror maker 2. When a topic without gaps is replicated, contiguous blocks of offsets are preserved. For example:

Say that MM2 mirrors 100 records, and emits two syncs: 0:100 and 100:200. We can detect when the gap between the upstream and downstream offsets is the same using subtraction, and then assume that 50:150 is also a valid translation. If the source topic has gaps, or goes through a restart, we should expect a discontinuity in the offset syncs, like 0:100 and 100:250 or 0:100 and 100:150.

This may allow us to restore much of the offset translation precision that was lost for simple contiguous topics, without additional memory usage, but at the risk of mis-translating some pathological situations when the source topic has gaps. This might be able to be enabled unconditionally, or enabled via a configuration.

Issue Links

• is related to: KAFKA-16364 MM2 High-Resolution Offset Translation