Details
-
Improvement
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
None
-
None
Description
MirrorMaker2 includes an offset translation feature which can translate the offsets for an upstream consumer group to a corresponding downstream consumer group. It does this by keeping a topic of offset-syncs to correlate upstream and downstream offsets, and translates any source offsets which are ahead of the replication flow.
However, if a replication flow is closer to the end of a topic than the consumer group, then the offset translation feature will refuse to translate the offset for correctness reasons. This is because the MirrorCheckpointTask only keeps the latest offset correlation between source and target, it does not have sufficient information to translate older offsets.
The workarounds for this issue are to:
1. Pause the replication flow occasionally to allow the source to get ahead of MM2
2. Increase the offset.lag.max to delay offset syncs, increasing the window for translation to happen. With the fix for KAFKA-12468, this will also increase the lag of applications that are ahead of the replication flow, so this is a tradeoff.
Instead, the MirrorCheckpointTask should provide correct and best-effort translation for consumer groups behind the replication flow by keeping additional state, or re-reading the offset-syncs topic. This should be a substantial improvement for use-cases where applications have a higher latency to commit than the replication flow, or where applications are reading from the earliest offset.
Attachments
Issue Links
- relates to
-
KAFKA-12468 Initial offsets are copied from source to target cluster
-
- Resolved
-
-
-
- Resolved
-
- links to