Critical Description
We have 3 node kafka cluster on RHEL.
We are getting following WARN messages on 2 nodes when using console conusmer/ console producer
Consumer Warning
[2020-04-08 06:05:02,356] WARN [Consumer clientId=consumer-1, groupId=console-consumer-5952] 1 partitions have leader brokers without a matching listener, including [DR27SAL_S_EVT_ACT-0] (org.apache.kafka.clients.NetworkClient)
Producer Warning
[2020-04-08 06:06:39,177] WARN [Producer clientId=console-producer] 2 partitions have leader brokers without a matching listener, including [FirstConsoleTopic-5, FirstConsoleTopic-2] (org.apache.kafka.clients.NetworkClient)
)
Got the following errors in one of our Kafka borker's Stage Change log
[2020-04-06 20:08:03,063] TRACE [Controller id=2 epoch=21] Received response {error_code=0} for request UPDATE_METADATA with correlation
id 2 sent to broker hostname.xxxxxx.xxx.com:9092 (id: 2 rack: null) (state.change.logger)
[2020-04-06 20:08:03,120] ERROR [Controller id=2 epoch=21] Controller 2 epoch 21 initiated state change of replica 1 for partition __cons
umer_offsets-22 from ReplicaDeletionIneligible to OfflineReplica failed (state.change.logger)
org.apache.zookeeper.KeeperException$SessionMovedException: KeeperErrorCode = Session moved for /controller_epoch
at org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at kafka.zk.KafkaZkClient$.kafka$zk$KafkaZkClient$$unwrapResponseWithControllerEpochCheck(KafkaZkClient.scala:1864)
at kafka.zk.KafkaZkClient.$anonfun$retryRequestsUntilConnected$2(KafkaZkClient.scala:1650)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:237)
at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at kafka.zk.KafkaZkClient.retryRequestsUntilConnected(KafkaZkClient.scala:1650)
at kafka.zk.KafkaZkClient.setTopicPartitionStatesRaw(KafkaZkClient.scala:204)
at kafka.zk.KafkaZkClient.updateLeaderAndIsr(KafkaZkClient.scala:261)
at kafka.controller.ZkReplicaStateMachine.doRemoveReplicasFromIsr(ReplicaStateMachine.scala:318)
at kafka.controller.ZkReplicaStateMachine.removeReplicasFromIsr(ReplicaStateMachine.scala:282)
at kafka.controller.ZkReplicaStateMachine.doHandleStateChanges(ReplicaStateMachine.scala:219)
at kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2(ReplicaStateMachine.scala:111)
at kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2$adapted(ReplicaStateMachine.scala:110)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:128)
at kafka.controller.ZkReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:110)
at kafka.controller.ReplicaStateMachine.startup(ReplicaStateMachine.scala:42)
at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:268)
at kafka.controller.KafkaController.elect(KafkaController.scala:1226)
at kafka.controller.KafkaController.processReelect(KafkaController.scala:1543)
at kafka.controller.KafkaController.process(KafkaController.scala:1584)
at kafka.controller.QueuedEvent.process(ControllerEventManager.scala:53)
at kafka.controller.ControllerEventManager$ControllerEventThread.$anonfun$doWork$1(ControllerEventManager.scala:137)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:137)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)