[KAFKA-2729] Cached zkVersion not equal to that in zookeeper, broker not recovering.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.8.2.1, 0.9.0.0, 0.10.0.0, 0.10.1.0, 0.11.0.0, 2.4.1
• Fix Version/s: 1.1.0
• Assignee: Onur Karaman
• Reporter: Danil Serdyuchenko
• Created: 03/Nov/15 15:00
• Updated: 23/Jul/22 16:14
• Resolved: 22/Dec/17 18:34

Description

After a small network wobble where zookeeper nodes couldn't reach each other, we started seeing a large number of undereplicated partitions. The zookeeper cluster recovered, however we continued to see a large number of undereplicated partitions. Two brokers in the kafka cluster were showing this in the logs:

[2015-10-27 11:36:00,888] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 (kafka.cluster.Partition)
[2015-10-27 11:36:00,891] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)

For all of the topics on the effected brokers. Both brokers only recovered after a restart. Our own investigation yielded nothing, I was hoping you could shed some light on this issue. Possibly if it's related to: https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 0.8.2.1.

Issue Links

• relates to: KAFKA-3042 updateIsr should stop after failed several times due to zkVersion issue
• relates to: KAFKA-3083 a soft failure in controller may leave a topic partition in an inconsistent state