[KAFKA-15144] MM2 Checkpoint downstreamOffset stuck to 1

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Bug
• Component/s: mirrormaker
• Assignee: Edoardo Comar
• Reporter: Edoardo Comar
• Created: 03/Jul/23 16:03
• Updated: 06/Jul/23 20:56
• Resolved: 04/Jul/23 13:18

Description

Steps to reproduce :

1.Start the source cluster

2.Start the target cluster

3.Start connect-mirror-maker.sh using a config like the attached

4.Create a topic in source cluster

5.produce a few messages

6.consume them all with autocommit enabled

 

7. then dump the Checkpoint topic content e.g.

% bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 --topic source.checkpoints.internal --from-beginning --formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter

Checkpoint{consumerGroupId=edogroup, topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=3, downstreamOffset=1, metadata=}

 

the downstreamOffset remains at 1, while, in a fresh cluster pair like with the source topic created while MM2 is running,
I'd expect the downstreamOffset to match the upstreamOffset.

Note that dumping the offset sync topic, shows matching initial offsets

% bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 --topic mm2-offset-syncs.source.internal --from-beginning --formatter org.apache.kafka.connect.mirror.formatters.OffsetSyncFormatter

OffsetSync{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=0, downstreamOffset=0}
 
 
 

Issue Links

• relates to: KAFKA-12468 Initial offsets are copied from source to target cluster