Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5027

Kafka Controller Redesign

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The goal of this redesign is to improve controller performance, controller maintainability, and cluster reliability.

      Documentation regarding what's being considered can be found here.

        Attachments

          Issue Links

          relates to

          Sub-task - The sub-task of the issue KAFKA-1120 Controller could miss a broker state change

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-6029 Controller should wait for the leader migration to finish before ack a ControlledShutdownRequest

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-3083 a soft failure in controller may leave a topic partition in an inconsistent state

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-4444 Aggregate requests sent from controller to broker during controlled shutdown

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-5296 Unable to write to some partitions of newly created topic in 10.2

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-3038 Speeding up partition reassignment after broker failure

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-3436 Speed up controlled shutdown.

          • Major - Major loss of function.
          • Resolved
          1.
          		 add controller integration tests Sub-task Resolved Onur Karaman
          2.
          		 convert kafka controller to a single-threaded event queue model Sub-task Resolved Onur Karaman
          3.
          		 remove preferred replica election state from ControllerContext Sub-task Resolved Onur Karaman
          4.
          		 add a tool analyzing zookeeper client performance across its various apis Sub-task Patch Available Onur Karaman
          5.
          		 move all partition and replica state transition rules into their states Sub-task Resolved Onur Karaman
          6.
          		 reset ControllerContext during resignation Sub-task Resolved Onur Karaman
          7.
          		 read current brokers from zookeeper upon processing broker change Sub-task Resolved Onur Karaman
          8.
          		 introduce async ZookeeperClient Sub-task Resolved Onur Karaman
          9.
          		 Use async ZookeeperClient in Controller Sub-task Resolved Onur Karaman
          10.
          		 cleanup javadocs and logging Sub-task In Progress Ismael Juma
          11.
          		 always leave the last surviving member of the ISR in ZK Sub-task Resolved Onur Karaman
          12.
          		 consider switching json parser from scala to jackson Sub-task Resolved Onur Karaman
          13.
          		 handle ZK session expiration properly when a new session can't be established Sub-task Resolved Prasanna Gautam
          14.
          		 Use async ZookeeperClient in SimpleAclAuthorizer Sub-task Resolved Manikumar
          15.
          		 Use async ZookeeperClient for DynamicConfigManager Sub-task Resolved Manikumar
          16.
          		 Use async ZookeeperClient for Admin operations Sub-task Resolved Manikumar
          17.
          		 add the notion of max inflight requests to async ZookeeperClient Sub-task Resolved Onur Karaman
          18.
          		 Add ZooKeeperRequestLatencyMs to KafkaZkClient Sub-task Resolved Ismael Juma
          19.
          		 Use ZookeeperClient in LogManager Sub-task Resolved Manikumar
          20.
          		 Use ZookeeperClient in GroupCoordinator and TransactionCoordinator Sub-task Resolved Manikumar
          21.
          		 Use ZookeeperClient in KafkaApis Sub-task Resolved Mickael Maison
          22.
          		 Use ZookeeperClient in ReplicaManager and Partition Sub-task Resolved Zhihong Yu
          23.
          		 Controller should wait for the leader migration to finish before ack a ControlledShutdownRequest Sub-task Resolved Zhanxiang (Patrick) Huang
          24.
          		 Controller could miss a broker state change Sub-task Resolved Zhanxiang (Patrick) Huang
          25.
          		 response error code checking Sub-task Open Unassigned
          26.
          		 consider fencing zookeeper updates with controller epoch zkVersion Sub-task Resolved Zhanxiang (Patrick) Huang
          27.
          		 Tests for KafkaZkClient Sub-task Resolved Sandor Murakozi
          28.
          		 minimize the number of triggers enqueuing PreferredReplicaLeaderElection events Sub-task Resolved Onur Karaman
          29.
          		 move ZK metrics in KafkaHealthCheck to ZookeeperClient Sub-task Resolved Jun Rao
          30.
          		 KafkaZkClient request queue time metric Sub-task Open Unassigned
          31.
          		 batch LeaderAndIsr requests during auto preferred leader election Sub-task Resolved huxihx

            Activity

              People

              • Assignee:
                onurkaraman Onur Karaman
                Reporter:
                onurkaraman Onur Karaman
                Reviewer:
                Jun Rao
              • Votes:
                9 Vote for this issue
                Watchers:
                35 Start watching this issue

                Dates

                • Created:
                  Updated: