Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3038

Speeding up partition reassignment after broker failure

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 0.9.0.0
    • None
    • controller, core
    • None

    Description

      After a broker failure the controller does several writes to Zookeeper for each partition on the failed broker. Writes are done one at a time, in closed loop, which is slow especially under high latency networks. Zookeeper has support for batching operations (the "multi" API). It is expected that substituting serial writes with batched ones should reduce failure handling time by an order of magnitude.

      This is identified as an issue in https://cwiki.apache.org/confluence/display/KAFKA/kafka+Detailed+Replication+Design+V3 (section End-to-end latency during a broker failure)

      Attachments

        Issue Links

          is part of

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-3210 Using asynchronous calls through the raw ZK API in ZkUtils

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-5027 Kafka Controller Redesign

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #2213

          Activity

            People

              Unassigned Unassigned
              enothereska Eno Thereska
              Flavio Paiva Junqueira Flavio Paiva Junqueira
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: