[KAFKA-5027] Kafka Controller Redesign

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Onur Karaman
• Reporter: Onur Karaman
• Reviewer: Jun Rao
• Created: 06/Apr/17 05:29
• Updated: 07/Dec/17 00:35

Description

The goal of this redesign is to improve controller performance, controller maintainability, and cluster reliability.

Documentation regarding what's being considered can be found here.

Issue Links

• relates to: KAFKA-1120 Controller could miss a broker state change
• relates to: KAFKA-6029 Controller should wait for the leader migration to finish before ack a ControlledShutdownRequest
• relates to: KAFKA-3083 a soft failure in controller may leave a topic partition in an inconsistent state
• relates to: KAFKA-4444 Aggregate requests sent from controller to broker during controlled shutdown
• relates to: KAFKA-5296 Unable to write to some partitions of newly created topic in 10.2
• relates to: KAFKA-3038 Speeding up partition reassignment after broker failure
• relates to: KAFKA-3436 Speed up controlled shutdown.

Sub-Tasks

• add a tool analyzing zookeeper client performance across its various apis: Patch Available
• cleanup javadocs and logging: In Progress
• response error code checking: Open
• KafkaZkClient request queue time metric: Open