Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10794

Replica leader election is too slow in the case of too many partitions

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.6.0, 2.5.1
    • 2.8.0
    • core
    • None

    Description

      There is more than 6000 topics and 300 brokers in my kafka cluster, and we frequently run kafka-preferred-replica-election.sh to rebalance our cluster. But the reblance process spendes too more time and cpu resource like the picture blow.

      We find that the function:'controllerContext.allPartitions' is invoked too many times.

       

      Attachments

        1. clipboard_image_1606806581629.png
          962 kB
          limeng
        2. image-2020-12-02-15-23-43-384.png
          203 kB
          limeng
        3. 截屏2020-12-01 下午2.36.19.png
          204 kB
          limeng
        4. 截屏2020-12-02 上午11.43.48.png
          313 kB
          limeng

        Issue Links

          Activity

            People

              monty limeng
              monty limeng
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: