[KAFKA-8708] Zookeeper Session expired either before or while waiting for connection

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.1
• Component/s: zkclient
• Assignee: Unassigned
• Reporter: Chethan Bheemaiah
• Created: 24/Jul/19 07:54
• Updated: 24/Jul/19 07:54

Description

Recently we had encountered an issue in one of our kafka cluster. One of the node went down and was not joining the kafka cluster on restart. We had observed Session expired error messages in server.log

Below is one message
ERROR kafka.common.ZkNodeChangeNotificationListener: Error while processing notification change for path = /config/changes
kafka.zookeeper.ZooKeeperClientExpiredException: Session expired either before or while waiting for connection
    at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:238)
    at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:226)
    at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:226)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
    at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:226)
    at kafka.zookeeper.ZooKeeperClient$$anonfun$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:220)
    at kafka.zookeeper.ZooKeeperClient$$anonfun$waitUntilConnected$1.apply(ZooKeeperClient.scala:220)
    at kafka.zookeeper.ZooKeeperClient$$anonfun$waitUntilConnected$1.apply(ZooKeeperClient.scala:220)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
    at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:219)
    at kafka.zk.KafkaZkClient.retryRequestsUntilConnected(KafkaZkClient.scala:1510)
    at kafka.zk.KafkaZkClient.kafka$zk$KafkaZkClient$$retryRequestUntilConnected(KafkaZkClient.scala:1486)
    at kafka.zk.KafkaZkClient.getChildren(KafkaZkClient.scala:585)
    at kafka.common.ZkNodeChangeNotificationListener.kafka$common$ZkNodeChangeNotificationListener$$processNotifications(ZkNodeChangeNotificationListener.scala:82)
    at kafka.common.ZkNodeChangeNotificationListener$ChangeNotification.process(ZkNodeChangeNotificationListener.scala:119)
    at kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread.doWork(ZkNodeChangeNotificationListener.scala:145)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)