Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9836

org.apache.zookeeper.KeeperException$SessionMovedException: KeeperErrorCode = Session moved for /controller_epoch

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0
    • Component/s: admin
    • Labels:
      None

      Description

      We have 3 node kafka cluster on RHEL.

      We are getting following WARN messages on 2 nodes when using console conusmer/ console producer 

      Consumer Warning

      [2020-04-08 06:05:02,356] WARN [Consumer clientId=consumer-1, groupId=console-consumer-5952] 1 partitions have leader brokers without a matching listener, including [DR27SAL_S_EVT_ACT-0] (org.apache.kafka.clients.NetworkClient)

      Producer Warning

      [2020-04-08 06:06:39,177] WARN [Producer clientId=console-producer] 2 partitions have leader brokers without a matching listener, including [FirstConsoleTopic-5, FirstConsoleTopic-2] (org.apache.kafka.clients.NetworkClient)

      )

       

      Got the following errors in one of our Kafka borker's Stage Change log

       [2020-04-06 20:08:03,063] TRACE [Controller id=2 epoch=21] Received response {error_code=0} for request UPDATE_METADATA with correlation
      id 2 sent to broker hostname.xxxxxx.xxx.com:9092 (id: 2 rack: null) (state.change.logger)

      [2020-04-06 20:08:03,120] ERROR [Controller id=2 epoch=21] Controller 2 epoch 21 initiated state change of replica 1 for partition __cons
      umer_offsets-22 from ReplicaDeletionIneligible to OfflineReplica failed (state.change.logger)
      org.apache.zookeeper.KeeperException$SessionMovedException: KeeperErrorCode = Session moved for /controller_epoch
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
      at kafka.zk.KafkaZkClient$.kafka$zk$KafkaZkClient$$unwrapResponseWithControllerEpochCheck(KafkaZkClient.scala:1864)
      at kafka.zk.KafkaZkClient.$anonfun$retryRequestsUntilConnected$2(KafkaZkClient.scala:1650)
      at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
      at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
      at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
      at scala.collection.TraversableLike.map(TraversableLike.scala:237)
      at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
      at scala.collection.AbstractTraversable.map(Traversable.scala:108)
      at kafka.zk.KafkaZkClient.retryRequestsUntilConnected(KafkaZkClient.scala:1650)
      at kafka.zk.KafkaZkClient.setTopicPartitionStatesRaw(KafkaZkClient.scala:204)
      at kafka.zk.KafkaZkClient.updateLeaderAndIsr(KafkaZkClient.scala:261)
      at kafka.controller.ZkReplicaStateMachine.doRemoveReplicasFromIsr(ReplicaStateMachine.scala:318)
      at kafka.controller.ZkReplicaStateMachine.removeReplicasFromIsr(ReplicaStateMachine.scala:282)
      at kafka.controller.ZkReplicaStateMachine.doHandleStateChanges(ReplicaStateMachine.scala:219)
      at kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2(ReplicaStateMachine.scala:111)
      at kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2$adapted(ReplicaStateMachine.scala:110)
      at scala.collection.immutable.Map$Map1.foreach(Map.scala:128)
      at kafka.controller.ZkReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:110)
      at kafka.controller.ReplicaStateMachine.startup(ReplicaStateMachine.scala:42)
      at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:268)
      at kafka.controller.KafkaController.elect(KafkaController.scala:1226)
      at kafka.controller.KafkaController.processReelect(KafkaController.scala:1543)
      at kafka.controller.KafkaController.process(KafkaController.scala:1584)
      at kafka.controller.QueuedEvent.process(ControllerEventManager.scala:53)
      at kafka.controller.ControllerEventManager$ControllerEventThread.$anonfun$doWork$1(ControllerEventManager.scala:137)
      at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
      at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
      at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:137)
      at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              KafkaJagadish Jagadish
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: