[KAFKA-3173] Error while moving some partitions to OnlinePartition state

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.9.0.0
• Assignee: Flavio Paiva Junqueira
• Reporter: Flavio Paiva Junqueira
• Reviewer: Jun Rao
• Created: 30/Jan/16 00:36
• Updated: 02/Jan/17 22:48

Description

We observed another instance of the problem reported in KAFKA-2300, but this time the error appeared in the partition state machine. In KAFKA-2300, we haven't cleaned up the state in PartitionStateMachine and ReplicaStateMachine as we do in KafkaController.

Here is the stack trace:

2016-01-29 15:26:51,393] ERROR [Partition state machine on Controller 0]: Error while moving some partitions to OnlinePartition state 
(kafka.controller.PartitionStateMachine)java.lang.IllegalStateException: Controller to broker state change requests batch is not empty while creating a new one. 
Some LeaderAndIsr state changes Map(0 -> Map(foo-0 -> (LeaderAndIsrInfo:
(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0))) might be lost        at 
kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:254)        at 
kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:144)        at 
kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:517)        at 
kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:504)        at 
kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(PartitionStateMachine.scala:437)        at 
kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)        at 
kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)        at 
kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)        at 
kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:418)        at 
org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842)        at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)

Issue Links

• is related to: KAFKA-2300 Error in controller log when broker tries to rejoin cluster
• is related to: KAFKA-2818 Clean up Controller Object on forced Resignation

Sub-Tasks

• Log exception message in ControllerBrokerRequestBatch: Resolved