[KAFKA-16297] Race condition while promoting future replica can lead to partition unavailability.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.7.0
• Fix Version/s: 3.8.0, 3.7.1
• Component/s: jbod
• Assignee: Igor Soarez
• Reporter: Igor Soarez
• Created: 22/Feb/24 13:48
• Updated: 10/Apr/24 09:57
• Resolved: 10/Apr/24 09:57

Description

KIP-858 proposed that when a directory failure occurs after changing the assignment of a replica that's moved between two directories in the same broker, but before the future replica promotion completes, the broker should reassign the replica to inform the controller of its correct status. But this hasn't yet been implemented, and without it this failure may lead to indefinite partition unavailability.

Example scenario:

1. A broker which leads partition P receives a request to alter the replica from directory A to directory B.
2. The broker creates a future replica in directory B and starts a replica fetcher.
3. Once the future replica first catches up, the broker queues a reassignment to inform the controller of the directory change.
4. The next time the replica catches up, the broker briefly blocks appends and promotes the replica. However, before the promotion is attempted, directory A fails.
5. The controller was informed that P in now in directory B before it received the notification that directory A has failed, so it does not elect a new leader, and as long as the broker is online, partition A remains unavailable.

 

 

Issue Links

• links to: GitHub Pull Request #15557