[KAFKA-13988] Mirrormaker 2 auto.offset.reset=latest not working

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.7.0, 3.6.2
• Component/s: mirrormaker
• Environment: Source Kafka cluster running on Ubuntu 20 ; Source Kafka cluster Kafka v0.10 ; ; Target Kafka cluster running in AWS MSK ; Target Kafka cluster Kafka v2.6.2 ; ; Mirrormaker version 3.2.0 running on Ubuntu 20.
• Assignee: Chris Egerton
• Reporter: Daniel Florek
• Created: 14/Jun/22 07:54
• Updated: 08/Feb/24 10:35
• Resolved: 04/Dec/23 21:38

Description

Hi. 

I have problem setting up mirroring with MM2 from latest offset between 2 clusters. In logs I can see that Consumer that is consuming topics has auto.offset.reset property set to latest. But still topics are read from offset 0. I am using following configuration:

 

clusters = A, B

A.bootstrap.servers = broker-01A:9092
B.bootstrap.servers = broker-01B:9092,broker-02B:9092,broker-03B:9092

replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy

#Enable replication between clusters and define topics which should be replicated
A->B.enabled = true
A->B.topics = .*
A->B.replication.factor=3
A->B.emit.heartbeats.enabled = true
A->B.emit.checkpoints.enabled = true

auto.offset.reset=latest
consumer.auto.offset.reset=latest
A.consumer.auto.offset.reset=latest
B.consumer.auto.offset.reset=latest
refresh.topics.enabled=true

heartbeats.topic.replication.factor=1
checkpoints.topic.replication.factor=1
offset-syncs.topic.replication.factor=1

config.storage.replication.factor = 1
offset.storage.replication.factor = 1
status.storage.replication.factor = 1 

I am using Kafka 3.2.0 for Mirrormaker 2. Source kafka cluster is 1 broker running on EC2 instance in AWS (quite an old version I think 0.10). Target kafka cluster contains 3 brokers running in AWS MSK (version 2.6.2). 

Could you point me what I am doing wrong? Or is this possibly a bug?

 

Issue Links

• is duplicated by: KAFKA-12937 Mirrormaker2 can only start from the beginning of a topic
• links to: GitHub Pull Request #12358
• links to: GitHub Pull Request #13905
• links to: GitHub Pull Request #14567