[KAFKA-9076] MirrorMaker 2.0 automated consumer offset sync

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.7.0
• Component/s: mirrormaker
• Labels: mirrormaker pull-request-available
• Assignee: Ning Zhang
• Reporter: Ning Zhang
• Created: 22/Oct/19 07:16
• Updated: 20/Jan/21 22:57
• Resolved: 26/Jun/20 15:17

Description

To calculate the translated consumer offset in the target cluster, currently `Mirror-client` provides a function called "remoteConsumerOffsets()" that is used by "RemoteClusterUtils" for one-time purpose.

In order to make the consumer and stream applications migrate from source to target cluster transparently and conveniently, e.g. in event of source cluster failure, a background job is proposed to periodically sync the consumer offsets from the source to target cluster, so that when the consumer and stream applications switch to the target cluster, it will resume to consume from where it left off at source cluster.

 KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0

https://github.com/apache/kafka/pull/7577

Issue Links

• links to: GitHub Pull Request #7577
• mentioned in: Page Loading...