[KAFKA-12635] Mirrormaker 2 offset sync is incorrect if the target partition is empty

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 3.3.0
• Component/s: mirrormaker
• Assignee: Mickael Maison
• Reporter: Frank Yi
• Reviewer: Ning Zhang
• Created: 08/Apr/21 22:56
• Updated: 23/Jan/23 14:15
• Resolved: 16/May/22 18:45

Description

This bug occurs when using Mirrormaker with "sync.group.offsets.enabled = true".

If a source partition is empty, but the source consumer group's offset for that partition is non-zero, then Mirrormaker sets the target consumer group's offset for that partition to the literal, not translated, offset of the source consumer group. This state can be reached if the source consumer group consumed some records that were now deleted (like by a retention policy), or if Mirrormaker replication is set to start at "latest". This bug causes the target consumer group's lag for that partition to be negative and breaks offset sync for that partition until lag is positive.

The correct behavior when the source partition is empty would be to set the target offset to the translated offset, not literal offset, which in this case would always be 0. 

Original email thread on this issue: https://lists.apache.org/thread.html/r7c54ee5f57227367b911d4abffa72781772d8dd3b72d75eb65ee19f7%40%3Cusers.kafka.apache.org%3E

Issue Links

• is duplicated by: KAFKA-13452 MM2 creates invalid checkpoint when offset mapping is not available
• links to: GitHub Pull Request #10644
• links to: GitHub Pull Request #11748