[KAFKA-5027] Kafka Controller Redesign

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Onur Karaman
• Reporter: Onur Karaman
• Reviewer: Jun Rao
• Created: 06/Apr/17 05:29
• Updated: 07/Dec/17 00:35

Description

The goal of this redesign is to improve controller performance, controller maintainability, and cluster reliability.

Documentation regarding what's being considered can be found here.

Issue Links

• relates to: KAFKA-1120 Controller could miss a broker state change
• relates to: KAFKA-6029 Controller should wait for the leader migration to finish before ack a ControlledShutdownRequest
• relates to: KAFKA-3083 a soft failure in controller may leave a topic partition in an inconsistent state
• relates to: KAFKA-4444 Aggregate requests sent from controller to broker during controlled shutdown
• relates to: KAFKA-5296 Unable to write to some partitions of newly created topic in 10.2
• relates to: KAFKA-3038 Speeding up partition reassignment after broker failure
• relates to: KAFKA-3436 Speed up controlled shutdown.

Sub-Tasks

• add controller integration tests: Resolved
• convert kafka controller to a single-threaded event queue model: Resolved
• remove preferred replica election state from ControllerContext: Resolved
• add a tool analyzing zookeeper client performance across its various apis: Patch Available
• move all partition and replica state transition rules into their states: Resolved
• reset ControllerContext during resignation: Resolved
• read current brokers from zookeeper upon processing broker change: Resolved
• introduce async ZookeeperClient: Resolved
• Use async ZookeeperClient in Controller: Resolved
• cleanup javadocs and logging: In Progress
• always leave the last surviving member of the ISR in ZK: Resolved
• consider switching json parser from scala to jackson: Resolved
• handle ZK session expiration properly when a new session can't be established: Resolved
• Use async ZookeeperClient in SimpleAclAuthorizer: Resolved
• Use async ZookeeperClient for DynamicConfigManager: Resolved
• Use async ZookeeperClient for Admin operations: Resolved
• add the notion of max inflight requests to async ZookeeperClient: Resolved
• Add ZooKeeperRequestLatencyMs to KafkaZkClient: Resolved
• Use ZookeeperClient in LogManager: Resolved
• Use ZookeeperClient in GroupCoordinator and TransactionCoordinator: Resolved
• Use ZookeeperClient in KafkaApis: Resolved
• Use ZookeeperClient in ReplicaManager and Partition: Resolved
• Controller should wait for the leader migration to finish before ack a ControlledShutdownRequest: Resolved
• Controller could miss a broker state change: Resolved
• response error code checking: Open
• consider fencing zookeeper updates with controller epoch zkVersion: Resolved
• Tests for KafkaZkClient: Resolved
• minimize the number of triggers enqueuing PreferredReplicaLeaderElection events: Resolved
• move ZK metrics in KafkaHealthCheck to ZookeeperClient: Resolved
• KafkaZkClient request queue time metric: Open
• batch LeaderAndIsr requests during auto preferred leader election: Resolved