2021-03-31 18:08:28,700 TRACE [Controller id=2 epoch=11] Sending become-leader LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640033, isr=[1, 0], zkVersion=640073, replicas=[1, 0], addingReplicas=[], removingReplicas=[], isNew=false) to broker 1 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,701 TRACE [Controller id=2 epoch=11] Sending become-follower LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640033, isr=[1, 0], zkVersion=640073, replicas=[1, 0], addingReplicas=[], removingReplicas=[], isNew=false) to broker 0 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,701 TRACE [Controller id=2 epoch=11] Sending StopReplica request StopReplicaPartitionState(partitionIndex=1, leaderEpoch=640032, deletePartition=false) to broker 2 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,701 TRACE [Controller id=2 epoch=11] Changed state of replica 2 for partition example-topic-1 from OfflineReplica to ReplicaDeletionStarted (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,701 TRACE [Controller id=2 epoch=11] Sending StopReplica request StopReplicaPartitionState(partitionIndex=1, leaderEpoch=640033, deletePartition=true) to broker 2 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,701 TRACE [Controller id=2 epoch=11] Changed state of replica 2 for partition example-topic-1 from ReplicaDeletionStarted to ReplicaDeletionSuccessful (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,701 TRACE [Broker id=2] Cached leader info UpdateMetadataPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640033, isr=[1, 0], zkVersion=640073, replicas=[1, 0], offlineReplicas=[]) for partition example-topic-1 in response to UpdateMetadata request sent by controller 2 epoch 11 with correlation id 72 (state.change.logger) [data-plane-kafka-request-handler-7]
2021-03-31 18:08:28,701 TRACE [Controller id=2 epoch=11] Changed state of replica 2 for partition example-topic-1 from ReplicaDeletionSuccessful to NonExistentReplica (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,702 TRACE [Broker id=2] Received StopReplica request StopReplicaPartitionState(partitionIndex=1, leaderEpoch=640032, deletePartition=false) correlation id 73 from controller 2 epoch 11 for partition example-topic-1 (state.change.logger) [data-plane-kafka-request-handler-3]
2021-03-31 18:08:28,702 INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions Set(example-topic-1) (kafka.server.ReplicaFetcherManager) [data-plane-kafka-request-handler-3]
2021-03-31 18:08:28,702 INFO [ReplicaAlterLogDirsManager on broker 2] Removed fetcher for partitions Set(example-topic-1) (kafka.server.ReplicaAlterLogDirsManager) [data-plane-kafka-request-handler-3]
2021-03-31 18:08:28,703 TRACE [Broker id=2] Received StopReplica request StopReplicaPartitionState(partitionIndex=1, leaderEpoch=640033, deletePartition=true) correlation id 74 from controller 2 epoch 11 for partition example-topic-1 (state.change.logger) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:28,703 INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions Set(example-topic-1) (kafka.server.ReplicaFetcherManager) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:28,703 INFO [ReplicaAlterLogDirsManager on broker 2] Removed fetcher for partitions Set(example-topic-1) (kafka.server.ReplicaAlterLogDirsManager) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:28,704 INFO [Controller id=2] Successfully updated assignment of partition example-topic-1 to ReplicaAssignment(replicas=1,0, addingReplicas=, removingReplicas=) (kafka.controller.KafkaController) [controller-event-thread]
2021-03-31 18:08:28,740 DEBUG [Controller id=2] Updating assignment of partition example-topic-1 from ReplicaAssignment(replicas=1,0, addingReplicas=, removingReplicas=) to ReplicaAssignment(replicas=1,2,0, addingReplicas=2, removingReplicas=) (kafka.controller.KafkaController) [controller-event-thread]
2021-03-31 18:08:28,743 INFO [Controller id=2] Successfully updated assignment of partition example-topic-1 to ReplicaAssignment(replicas=1,2,0, addingReplicas=2, removingReplicas=) (kafka.controller.KafkaController) [controller-event-thread]
2021-03-31 18:08:28,744 DEBUG [Controller id=2] Updating leader epoch for partition example-topic-1 (kafka.controller.KafkaController) [controller-event-thread]
2021-03-31 18:08:28,748 INFO [Controller id=2] Updated leader epoch for partition example-topic-1 to 640034 (kafka.controller.KafkaController) [controller-event-thread]
2021-03-31 18:08:28,748 TRACE [Controller id=2 epoch=11] Sending become-follower LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640034, isr=[1, 0], zkVersion=640074, replicas=[1, 2, 0], addingReplicas=[2], removingReplicas=[], isNew=true) to broker 2 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,749 TRACE [Controller id=2 epoch=11] Sending become-leader LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640034, isr=[1, 0], zkVersion=640074, replicas=[1, 2, 0], addingReplicas=[2], removingReplicas=[], isNew=false) to broker 1 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,749 TRACE [Controller id=2 epoch=11] Sending become-follower LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640034, isr=[1, 0], zkVersion=640074, replicas=[1, 2, 0], addingReplicas=[2], removingReplicas=[], isNew=false) to broker 0 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,749 INFO [Controller id=2 epoch=11] Sent LeaderAndIsr request (Leader:1,ISR:1,0,LeaderEpoch:640034,ControllerEpoch:11) with new replica assignment ReplicaAssignment(replicas=1,2,0, addingReplicas=2, removingReplicas=) to leader 1 for partition being reassigned example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,772 TRACE [Controller id=2 epoch=11] Changed state of replica 2 for partition example-topic-1 from NonExistentReplica to NewReplica (state.change.logger) [controller-event-thread]
2021-03-31 18:08:28,772 TRACE [Controller id=2 epoch=11] Sending become-follower LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640034, isr=[1, 0], zkVersion=640074, replicas=[1, 2, 0], addingReplicas=[2], removingReplicas=[], isNew=true) to broker 2 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:29,024 INFO Log for partition example-topic-1 is renamed to /var/lib/kafka/data-0/kafka-log2/example-topic-1.7f4059549790425e8e393d3b4c4c1236-delete and is scheduled for deletion (kafka.log.LogManager) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,032 TRACE [Broker id=2] Cached leader info UpdateMetadataPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640033, isr=[1, 0], zkVersion=640073, replicas=[1, 0], offlineReplicas=[]) for partition example-topic-1 in response to UpdateMetadata request sent by controller 2 epoch 11 with correlation id 75 (state.change.logger) [data-plane-kafka-request-handler-4]
2021-03-31 18:08:29,316 TRACE [Broker id=2] Received LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640034, isr=[1, 0], zkVersion=640074, replicas=[1, 2, 0], addingReplicas=[2], removingReplicas=[], isNew=true) correlation id 81 from controller 2 epoch 11 (state.change.logger) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,316 TRACE [Broker id=2] Handling LeaderAndIsr request correlationId 81 from controller 2 epoch 11 starting the become-follower transition for partition example-topic-1 with leader 1 (state.change.logger) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,319 INFO [Log partition=example-topic-1, dir=/var/lib/kafka/data-0/kafka-log2] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,319 INFO Created log for partition example-topic-1 in /var/lib/kafka/data-0/kafka-log2/example-topic-1 with properties {compression.type -> producer, min.insync.replicas -> 2, message.downconversion.enable -> true, segment.jitter.ms -> 0, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, retention.ms -> 604800000, segment.bytes -> 1073741824, flush.messages -> 9223372036854775807, message.format.version -> 2.7-IV2, max.compaction.lag.ms -> 9223372036854775807, file.delete.delay.ms -> 60000, max.message.bytes -> 1048588, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, preallocate -> false, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, segment.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760}. (kafka.log.LogManager) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,320 INFO [Partition example-topic-1 broker=2] No checkpointed highwatermark is found for partition example-topic-1 (kafka.cluster.Partition) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,320 INFO [Partition example-topic-1 broker=2] Log loaded for partition example-topic-1 with initial high watermark 0 (kafka.cluster.Partition) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,320 INFO [Broker id=2] Follower example-topic-1 starts at leader epoch 640034 from offset 0 with high watermark 0. Previous leader epoch was -1. (state.change.logger) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,320 INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions Set(example-topic-1) (kafka.server.ReplicaFetcherManager) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,320 INFO [ReplicaFetcherManager on broker 2] Added fetcher to broker 1 for partitions Map(example-topic-1 -> (offset=0, leaderEpoch=640034)) (kafka.server.ReplicaFetcherManager) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,320 TRACE [Broker id=2] Completed LeaderAndIsr request correlationId 81 from controller 2 epoch 11 for the become-follower transition for partition example-topic-1 with leader 1 (state.change.logger) [data-plane-kafka-request-handler-6]
2021-03-31 18:08:29,321 TRACE [Broker id=2] Cached leader info UpdateMetadataPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640034, isr=[1, 0], zkVersion=640074, replicas=[1, 2, 0], offlineReplicas=[]) for partition example-topic-1 in response to UpdateMetadata request sent by controller 2 epoch 11 with correlation id 82 (state.change.logger) [data-plane-kafka-request-handler-4]
2021-03-31 18:08:29,321 TRACE [Broker id=2] Received LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640034, isr=[1, 0], zkVersion=640074, replicas=[1, 2, 0], addingReplicas=[2], removingReplicas=[], isNew=true) correlation id 83 from controller 2 epoch 11 (state.change.logger) [data-plane-kafka-request-handler-5]
2021-03-31 18:08:29,321 INFO [Broker id=2] Ignoring LeaderAndIsr request from controller 2 with correlation id 83 epoch 11 for partition example-topic-1 since its associated leader epoch 640034 matches the current leader epoch (state.change.logger) [data-plane-kafka-request-handler-5]
2021-03-31 18:08:29,322 TRACE [Broker id=2] Cached leader info UpdateMetadataPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640034, isr=[1, 0], zkVersion=640074, replicas=[1, 2, 0], offlineReplicas=[]) for partition example-topic-1 in response to UpdateMetadata request sent by controller 2 epoch 11 with correlation id 84 (state.change.logger) [data-plane-kafka-request-handler-0]
2021-03-31 18:08:29,551 INFO [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Truncating partition example-topic-1 to local high watermark 0 (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
2021-03-31 18:08:29,551 INFO [Log partition=example-topic-1, dir=/var/lib/kafka/data-0/kafka-log2] Truncating to 0 has no effect as the largest offset in the log is -1 (kafka.log.Log) [ReplicaFetcherThread-0-1]
2021-03-31 18:08:29,556 WARN [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Reset fetch offset for partition example-topic-1 from 0 to current leader's start offset 483796 (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
2021-03-31 18:08:29,556 INFO The cleaning for partition example-topic-1 is aborted and paused (kafka.log.LogManager) [ReplicaFetcherThread-0-1]
2021-03-31 18:08:29,556 INFO [Log partition=example-topic-1, dir=/var/lib/kafka/data-0/kafka-log2] Deleting segments as part of log truncation: LogSegment(baseOffset=0, size=0, lastModifiedTime=1617214109317, largestRecordTimestamp=None) (kafka.log.Log) [ReplicaFetcherThread-0-1]
2021-03-31 18:08:29,560 INFO Cleaning for partition example-topic-1 is resumed (kafka.log.LogManager) [ReplicaFetcherThread-0-1]
2021-03-31 18:08:29,563 INFO [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Current offset 0 for partition example-topic-1 is out of range, which typically implies a leader change. Reset fetch offset to 483796 (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
2021-03-31 18:08:33,739 DEBUG [Controller id=2] Updating leader epoch for partition example-topic-1 (kafka.controller.KafkaController) [controller-event-thread]
2021-03-31 18:08:33,744 INFO [Controller id=2] Updated leader epoch for partition example-topic-1 to 640035 (kafka.controller.KafkaController) [controller-event-thread]
2021-03-31 18:08:33,744 TRACE [Controller id=2 epoch=11] Sending become-follower LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640035, isr=[1, 0], zkVersion=640075, replicas=[1, 2, 0], addingReplicas=[2], removingReplicas=[], isNew=true) to broker 2 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:33,744 TRACE [Controller id=2 epoch=11] Sending become-leader LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640035, isr=[1, 0], zkVersion=640075, replicas=[1, 2, 0], addingReplicas=[2], removingReplicas=[], isNew=false) to broker 1 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:33,744 TRACE [Controller id=2 epoch=11] Sending become-follower LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640035, isr=[1, 0], zkVersion=640075, replicas=[1, 2, 0], addingReplicas=[2], removingReplicas=[], isNew=false) to broker 0 for partition example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:33,744 TRACE [Broker id=2] Received LeaderAndIsr request LeaderAndIsrPartitionState(topicName='example-topic', partitionIndex=1, controllerEpoch=11, leader=1, leaderEpoch=640035, isr=[1, 0], zkVersion=640075, replicas=[1, 2, 0], addingReplicas=[2], removingReplicas=[], isNew=true) correlation id 93 from controller 2 epoch 11 (state.change.logger) [data-plane-kafka-request-handler-2]
2021-03-31 18:08:33,745 INFO [Controller id=2 epoch=11] Sent LeaderAndIsr request (Leader:1,ISR:1,0,LeaderEpoch:640035,ControllerEpoch:11) with new replica assignment ReplicaAssignment(replicas=1,2,0, addingReplicas=2, removingReplicas=) to leader 1 for partition being reassigned example-topic-1 (state.change.logger) [controller-event-thread]
2021-03-31 18:08:33,745 TRACE [Broker id=2] Handling LeaderAndIsr request correlationId 93 from controller 2 epoch 11 starting the become-follower transition for partition example-topic-1 with leader 1 (state.change.logger) [data-plane-kafka-request-handler-2]
2021-03-31 18:08:33,745 INFO [Broker id=2] Follower example-topic-1 starts at leader epoch 640035 from offset 483796 with high watermark 483796. Previous leader epoch was 640034. (state.change.logger) [data-plane-kafka-request-handler-2]
2021-03-31 18:08:33,745 ERROR [Controller id=2 epoch=11] Controller 2 epoch 11 initiated state change of replica 2 for partition example-topic-1 from NewReplica to NewReplica failed (state.change.logger) [controller-event-thread]
java.lang.IllegalStateException: Replica [Topic=example-topic,Partition=1,Replica=2] should be in the NonExistentReplica states before moving to NewReplica state. Instead it is in NewReplica state
        at kafka.controller.ZkReplicaStateMachine.logInvalidTransition(ReplicaStateMachine.scala:435)
        at kafka.controller.ZkReplicaStateMachine.$anonfun$doHandleStateChanges$2(ReplicaStateMachine.scala:164)
        at kafka.controller.ZkReplicaStateMachine.$anonfun$doHandleStateChanges$2$adapted(ReplicaStateMachine.scala:164)
        at scala.collection.immutable.List.foreach(List.scala:431)
        at kafka.controller.ZkReplicaStateMachine.doHandleStateChanges(ReplicaStateMachine.scala:164)
        at kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2(ReplicaStateMachine.scala:112)
        at kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2$adapted(ReplicaStateMachine.scala:111)
        at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.compat.MapExtensionMethods$.$anonfun$foreachEntry$1(PackageShared.scala:431)
        at scala.collection.immutable.Map$Map1.foreach(Map.scala:192)
        at kafka.controller.ZkReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:111)
        at kafka.controller.KafkaController.$anonfun$startNewReplicasForReassignedPartition$1(KafkaController.scala:1045)
        at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at kafka.controller.KafkaController.startNewReplicasForReassignedPartition(KafkaController.scala:1044)
        at kafka.controller.KafkaController.onPartitionReassignment(KafkaController.scala:736)
        at kafka.controller.KafkaController.$anonfun$maybeTriggerPartitionReassignment$1(KafkaController.scala:826)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
        at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
        at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
        at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
        at scala.collection.TraversableLike.map(TraversableLike.scala:286)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at kafka.controller.KafkaController.maybeTriggerPartitionReassignment(KafkaController.scala:816)
        at kafka.controller.KafkaController.processApiPartitionReassignment(KafkaController.scala:1794)
        at kafka.controller.KafkaController.process(KafkaController.scala:2420)
        at kafka.controller.QueuedEvent.process(ControllerEventManager.scala:52)
        at kafka.controller.ControllerEventManager$ControllerEventThread.process$1(ControllerEventManager.scala:130)
        at kafka.controller.ControllerEventManager$ControllerEventThread.$anonfun$doWork$1(ControllerEventManager.scala:133)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
        at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:133)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)