[KAFKA-3228] Partition reassignment failure for brokers freshly added to cluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.8.2.1
• Component/s: controller
• Assignee: Neha Narkhede
• Reporter: Andrew Olson
• Created: 11/Feb/16 15:03
• Updated: 05/Dec/16 14:11
• Resolved: 05/Dec/16 12:16

Description

After adding about new 20 brokers to double the size of an existing production Kafka deployment, when attempting to rebalance partitions we were initially unable to reassign any partitions to 5 of the 20. There was no problem with the other 15. The controller broker logged error messages like:

ERROR kafka.controller.KafkaController: [Controller 19]: Error completing reassignment of partition [TOPIC-NAME,2]
kafka.common.KafkaException: Only 4,33 replicas out of the new set of replicas 4,34,33 for partition [TOPIC-NAME,2]
to be reassigned are alive. Failing partition reassignment
	at kafka.controller.KafkaController.initiateReassignReplicasForTopicPartition(KafkaController.scala:611)
	at kafka.controller.PartitionsReassignedListener$$anonfun$handleDataChange$4$$anonfun$apply$6.apply$mcV$sp(KafkaController.scala:1203)
	at kafka.controller.PartitionsReassignedListener$$anonfun$handleDataChange$4$$anonfun$apply$6.apply(KafkaController.scala:1197)
	at kafka.controller.PartitionsReassignedListener$$anonfun$handleDataChange$4$$anonfun$apply$6.apply(KafkaController.scala:1197)
	at kafka.utils.Utils$.inLock(Utils.scala:535)
	at kafka.controller.PartitionsReassignedListener$$anonfun$handleDataChange$4.apply(KafkaController.scala:1196)
	at kafka.controller.PartitionsReassignedListener$$anonfun$handleDataChange$4.apply(KafkaController.scala:1195)
	at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
	at kafka.controller.PartitionsReassignedListener.handleDataChange(KafkaController.scala:1195)
	at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:751)
	at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)

We reattempted the reassignment to one of these new brokers, with the same result.

We also saw these messages in the controller's log. There was a "Broken pipe" error for each of the new brokers.

2016-02-09 12:13:22,082 WARN kafka.controller.RequestSendThread: [Controller-19-to-broker-34-send-thread],
Controller 19 epoch 28 fails to send request Name:UpdateMetadataRequest...
java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
	at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
	at sun.nio.ch.IOUtil.write(IOUtil.java:148)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
	at java.nio.channels.SocketChannel.write(SocketChannel.java:502)
	at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:56)
	at kafka.network.Send$class.writeCompletely(Transmission.scala:75)
	at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:26)
	at kafka.network.BlockingChannel.send(BlockingChannel.scala:103)
	at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
	at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

WARN kafka.controller.RequestSendThread: [Controller-19-to-broker-34-send-thread],
Controller 19 epoch 28 fails to send request Name:UpdateMetadataRequest... to broker id:34...
Reconnecting to broker.
java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
	at kafka.utils.Utils$.read(Utils.scala:381)
	at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
	at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
	at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
	at kafka.network.BlockingChannel.receive(BlockingChannel.scala:111)
	at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:133)
	at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

INFO kafka.controller.RequestSendThread: [Controller-19-to-broker-34-send-thread], Controller 19 connected
to id:34... for sending state change requests

There were no error messages in the new broker log files, just the normal startup logs. A jstack did not reveal anything unusual with the threads, and using netstat the network connections looked normal.

We're running version 0.8.2.1. The new brokers were simultaneously started using a broadcast-style command. However we also had the same issue with a different Kafka cluster after starting up the new brokers individually about 30 seconds apart.

After stopping and restarting the 5 problematic new brokers, the reassignment was then successful, and they are now functioning normally.

Issue Links

• duplicates: KAFKA-4214 kafka-reassign-partitions fails all the time when brokers are bounced during reassignment