[KAFKA-9118] LogDirFailureHandler shouldn't use Zookeeper

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: David Arthur
• Reporter: Viktor Somogyi-Vass
• Created: 30/Oct/19 13:13
• Updated: 17/May/22 17:06

Description

As described in KIP-112:

2. A log directory stops working on a broker during runtime

- The controller watches the path /log_dir_event_notification for new znode.
- The broker detects offline log directories during runtime.
- The broker takes actions as if it has received StopReplicaRequest for this replica. More specifically, the replica is no longer considered leader and is removed from any replica fetcher thread. (The clients will receive a UnknownTopicOrPartitionException at this point)
- The broker notifies the controller by creating a sequential znode under path /log_dir_event_notification with data of the format {"version" : 1, "broker" : brokerId, "event" : LogDirFailure}.
- The controller reads the znode to get the brokerId and finds that the event type is LogDirFailure.
- The controller deletes the notification znode
- The controller sends LeaderAndIsrRequest to that broker to query the state of all topic partitions on the broker. The LeaderAndIsrResponse from this broker will specify KafkaStorageException for those partitions that are on the bad log directories.
- The controller updates the information of offline replicas in memory and trigger leader election as appropriate.
- The controller removes offline replicas from ISR in the ZK and sends LeaderAndIsrRequest with updated ISR to be used by partition leaders.
- The controller propagates the information of offline replicas to brokers by sending UpdateMetadataRequest.

Instead of the notification ZNode we should use a Kafka protocol that sends a notification message to the controller with the offline partitions. The controller then updates the information of offline replicas in memory and trigger leader election, then removes the replicas from ISR in ZK and sends a LAIR and an UpdateMetadataRequest.

Issue Links

• is related to: KAFKA-9837 New RPC for notifying controller of failed replica
• requires: KAFKA-9167 Implement a broker to controller request channel