Kafka
  1. Kafka
  2. KAFKA-890

The list of brokers for fetching metadata should be shuffled

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:

      Description

      The list of brokers in the metadata request is never shuffled. Which means that if some clients are not using a VIP for metadata requests, the first broker ends up servicing most metadata requests, leaving imbalanced load on the brokers. This issue is even more pronounced when there are several thousand clients talking to a cluster each using a broker list to fetch metadata.

      1. kafka-890.patch
        3 kB
        Neha Narkhede

        Activity

        Neha Narkhede created issue -
        Neha Narkhede made changes -
        Field Original Value New Value
        Status Open [ 1 ] In Progress [ 3 ]
        Hide
        Neha Narkhede added a comment -

        This patch shuffles the list of brokers used by both producer and consumer to fetch metadata.

        Show
        Neha Narkhede added a comment - This patch shuffles the list of brokers used by both producer and consumer to fetch metadata.
        Neha Narkhede made changes -
        Attachment kafka-890.patch [ 12581107 ]
        Neha Narkhede made changes -
        Status In Progress [ 3 ] Patch Available [ 10002 ]
        Hide
        Joel Koshy added a comment -

        +1

        It is worth noting that this is useful even in the presence of a VIP since the consumers don't currently use a VIP to to look up metadata.

        Show
        Joel Koshy added a comment - +1 It is worth noting that this is useful even in the presence of a VIP since the consumers don't currently use a VIP to to look up metadata.
        Neha Narkhede made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Neha Narkhede made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Neha Narkhede
            Reporter:
            Neha Narkhede
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development