[KAFKA-3848] Mirror Maker sometimes fails to mirror first batch of messages

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0.1, 0.10.0.0
• Component/s: mirrormaker
• Assignee: Unassigned
• Reporter: James Clarke
• Created: 15/Jun/16 19:45
• Updated: 02/Aug/18 19:54

Description

I am seeing an intermittent issue in Mirror Maker where the first batch of messages are not always mirrored to the target cluster. All messages after the first batch are mirrored.

I have a github repo (jc/kafka-mirror-maker-test) which reproduces the issue using Confluent's docker containers (running 0.10.0). However on our environment we are using our own kafka containers running 0.9.0.1.

Environment:

• edge datacenter dc1. 1 zk server, 1 kafka server.
• aggregate datacenter dc2. 1 zk server, 1 kafka server, 1 mirror maker.
• kafka server setup to auto create topics
• Mirrror maker configured mirror from dc1 to dc2 using a whitelist containing both explicitly topics and regex topics.

Steps to reproduce:

• Send message to a non-existent topic in dc1.
• Send a second message to topic in dc1.

Observed:

• After first message the topic is not created in dc2.
• After second message topic is present in dc2.
• Consuming from topic in dc1 shows both messages.
• Consuming from topic in dc2 shows only the second message.