[KAFKA-13452] MM2 creates invalid checkpoint when offset mapping is not available

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: mirrormaker
• Assignee: Viktor Somogyi-Vass
• Reporter: Daniel Urban
• Created: 12/Nov/21 14:25
• Updated: 28/Apr/22 13:43
• Resolved: 28/Apr/22 13:43

Description

MM2 checkpointing reads the offset-syncs topic to create offset mappings for committed consumer group offsets. In some corner cases, it is possible that a mapping is not available in offset-syncs - in that case, MM2 simply copies the source offset, which might not be a valid offset in the replica topic at all.

One possible situation is if there is an empty topic in the source cluster with a non-zero endoffset (e.g. retention already removed the records), and a consumer group which has a committed offset set to the end offset. If replication is configured to start replicating this topic, it will not have an offset mapping available in offset-syncs (as the topic is empty), causing MM2 to copy the source offset.

This can cause issues when auto offset sync is enabled, as the consumer group offset can be potentially set to a high number. MM2 never rewinds these offsets, so even when there is a correct offset mapping available, the offset will not be updated correctly.

Issue Links

• duplicates: KAFKA-12635 Mirrormaker 2 offset sync is incorrect if the target partition is empty
• links to: GitHub Pull Request #11492