Kafka
  1. Kafka
  2. KAFKA-161

Producer using broker list does not load balance requests across multiple partitions on a broker

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7
    • Component/s: None
    • Labels:
      None

      Description

      https://issues.apache.org/jira/browse/KAFKA-129 introduced a bug in the load balancing logic of the Producer using broker.list.
      Since the broker.list doesn't specify the number of partitions in total, it should ideally pick a broker randomly, and then send the produce request with partition id -1, so that the EventHandler routes the request to a random partition.
      Instead of that, it defaults to 1 partition on each broker and ends up using the Partitioner to pick a partitions amongst the available ones.

      1. KAFKA-161.patch
        8 kB
        Neha Narkhede
      2. KAFKA-161.patch
        6 kB
        Neha Narkhede

        Activity

        Hide
        Neha Narkhede added a comment -

        Fixing the load balancing strategy for the broker.list option on the Producer. With this change, when broker.list is used, the producer will pick a random broker and send request to partition -1 on that broker.

        Show
        Neha Narkhede added a comment - Fixing the load balancing strategy for the broker.list option on the Producer. With this change, when broker.list is used, the producer will pick a random broker and send request to partition -1 on that broker.
        Hide
        Jun Rao added a comment -

        1. We should guard logger.debug with the isDebugEnabled check.
        2. The method name getNumPartitionsForTopic is misleading since the return value is a list of partitions, instead of number of partitions. Can we rename it to sth like getPartitionListForTopic? Ditto for the name of the variable being assigned to.

        Show
        Jun Rao added a comment - 1. We should guard logger.debug with the isDebugEnabled check. 2. The method name getNumPartitionsForTopic is misleading since the return value is a list of partitions, instead of number of partitions. Can we rename it to sth like getPartitionListForTopic? Ditto for the name of the variable being assigned to.
        Hide
        Neha Narkhede added a comment -

        1. Guarded the logger.debug
        2. Changed the names of variables and functions to make more sense

        Show
        Neha Narkhede added a comment - 1. Guarded the logger.debug 2. Changed the names of variables and functions to make more sense
        Hide
        Jun Rao added a comment -

        +1 on the new patch.

        Show
        Jun Rao added a comment - +1 on the new patch.

          People

          • Assignee:
            Neha Narkhede
            Reporter:
            Neha Narkhede
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development