[KAFKA-1387] Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.8.1.1
• Fix Version/s: 0.9.0.0
• Labels: newbie patch zkclient-problems
• Assignee: Flavio Paiva Junqueira
• Reporter: Fedor Korotkiy
• Reviewer: Guozhang Wang
• Created: 10/Apr/14 19:48
• Updated: 24/Sep/15 17:15
• Resolved: 24/Sep/15 17:15

Description

Kafka broker re-registers itself in zookeeper every time handleNewSession() callback is invoked.

https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/server/KafkaHealthcheck.scala

Now imagine the following sequence of events.
1) Zookeeper session reestablishes. handleNewSession() callback is queued by the zkClient, but not invoked yet.
2) Zookeeper session reestablishes again, queueing callback second time.
3) First callback is invoked, creating /broker/[id] ephemeral path.
4) Second callback is invoked and it tries to create /broker/[id] path using createEphemeralPathExpectConflictHandleZKBug() function. But the path is already exists, so createEphemeralPathExpectConflictHandleZKBug() is getting stuck in the infinite loop.

Seems like controller election code have the same issue.

I'am able to reproduce this issue on the 0.8.1 branch from github using the following configs.

1. zookeeper
   tickTime=10
   dataDir=/tmp/zk/
   clientPort=2101
   maxClientCnxns=0

1. kafka
   broker.id=1
   log.dir=/tmp/kafka
   zookeeper.connect=localhost:2101

zookeeper.connection.timeout.ms=100
zookeeper.sessiontimeout.ms=100

Just start kafka and zookeeper and then pause zookeeper several times using Ctrl-Z.

Issue Links

• depends upon: ZOOKEEPER-1740 Zookeeper 3.3.4 loses ephemeral nodes under stress
• is related to: KAFKA-1451 Broker stuck due to leader election race