[KAFKA-3143] inconsistent state in ZK when all replicas are dead

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: controller
• Labels: reliability
• Assignee: Ismael Juma
• Reporter: Jun Rao
• Created: 24/Jan/16 20:19
• Updated: 11/Jul/19 15:53
• Resolved: 21/May/19 15:20

Description

This issue can be recreated in the following steps.

1. Start 3 brokers, 1, 2 and 3.
2. Create a topic with a single partition and 2 replicas, say on broker 1 and 2.
If we stop both replicas 1 and 2, depending on where the controller is, the leader and isr stored in ZK in the end are different.

If the controller is on broker 3, what's stored in ZK will be -1 for leader and an empty set for ISR.

On the other hand, if the controller is on broker 2 and we stop broker 1 followed by broker 2, what's stored in ZK will be 2 for leader and 2 for ISR.

The issue is that in the first case, the controller will call ReplicaStateMachine to transition to OfflineReplica, which will change the leader and isr. However, in the second case, the controller fails over, but we don't transition ReplicaStateMachine to OfflineReplica during controller initialization.

Issue Links

• duplicates: KAFKA-8152 Offline partition state not propagated by controller
• links to: GitHub Pull Request #5041