[KAFKA-1407] Broker can not return to ISR because of BadVersionException

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 0.8.1, 2.4.1
• Component/s: controller
• Assignee: Neha Narkhede
• Reporter: Dmitry Bugaychenko
• Created: 21/Apr/14 06:11
• Updated: 09/Oct/24 09:18
• Resolved: 09/Oct/24 09:18

Description

Each morning we found a broker out of ISR at stuck with log full of messages:

INFO   | jvm 1    | 2014/04/21 08:36:21 | [2014-04-21 09:36:21,907] ERROR Conditional update of path /brokers/topics/topic2/partitions/1/state with data {"controller_epoch":46,"leader":2,"version":1,"leader_epoch":38,"isr":[2]} and expected version 53 failed due to org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /brokers/topics/topic2/partitions/1/state (kafka.utils.ZkUtils$)
INFO   | jvm 1    | 2014/04/21 08:36:21 | [2014-04-21 09:36:21,907] INFO Partition [topic2,1] on broker 2: Cached zkVersion [53] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)

It seems that it can not recover after short netwrok break down and the only way to return it is restart it using kill -9.

Issue Links

• relates to: KAFKA-3042 updateIsr should stop after failed several times due to zkVersion issue