[KAFKA-7629] Mirror maker goes into infinite loop

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0
• Component/s: mirrormaker
• Environment: local
• Assignee: Unassigned
• Reporter: Darshan Mehta
• Created: 14/Nov/18 17:35
• Updated: 14/Nov/18 18:07

Description

Setup:

I have 2 kafka images running Spoify Kafka image https://hub.docker.com/r/spotify/kafka

Config:

Image 1:

• host: kafka1
• zk port : 2181
• broker port : 9092

Image 2:

• host: kafka2
• zk port : 1181
• broker port : 8092

Producer Properties for Mirror maker: 

bootstrap.servers=kafka2:8092

Consumer Properties for Mirror maker: 

bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true

 

Steps to replicate :

1. Start mirror maker with following command : 

   $KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config <producer_config_location> --consumer.config <consumer_config_location> --num.streams 1 --whitelist topic-1
   

1. Start local kafka console consumer to listen to topic-1 for kafka2:8092 

   $KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server kafka2:8092 --topic topic-1
   

1. Produce an event to kafka1:9092 - topic-1  -> It will be printed on the console in Step 2
2. Stop mirror maker with ctrl+C (started in step 1)
3. Restart mirror maker with same command
4. Produce an event onto the same topic (i.e. repeat step 3)
5. Both source and destination will be flooded with the same messages until mirror maker is stopped

Surprisingly, source kafka also gets flooded with the same message. I believe when restarted, the mirror maker is unable to read the state?