When a broker get bounced, ideally the sequence should be like this:
1.1. Broker shutdown resources.
1.2. Broker close zkClient (this will cause the ephemeral node of /brokers/ids/BROKER_ID to be deleted)
1.3. Broker restart and load the log segment
1.4. Broker create ephemeral node /brokers/ids/BROKER_ID
The broker side log s are:
On the controller side, the sequence should be:
2.1. Controlled shutdown the broker
2.2. BrokerChangeListener fired for /brokers/ids child change because ephemeral node is deleted in step 1.2
2.3. BrokerChangeListener fired again for /borkers/ids child change because the ephemeral node is created in 1.4
The issue I saw was on controller side, the broker change listener only fired once after step 1.4. So the controller did not see any broker change.
From ZK transaction log, the zk session in step 1.4 has already be closed.
In this case, there are 10 seconds between step 1.2 and step 1.4
ZK session timeout was set to 12 seconds.
According to our test, the ephemeral node in zookeeper will be deleted after the session is explicitly closed. But it seems not the case when the broker shuts down.
Another suspicious thing is that even after socket server one broker has shutdown, the controller was still be able to connect to the broker and send request.
Currently we are setting the zk session timeout to 6 seconds and this seems solve the problem. My hunch is that zookeeper somehow did not fire the callback in step 1.2. So step 2.2 was not triggered.
From the available log, the missing piece here is how do we know if the zk watcher has been fired by zookeeper in step 2.2?