Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-161

Producer using broker list does not load balance requests across multiple partitions on a broker

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7
    • Component/s: None
    • Labels:
      None

      Description

      https://issues.apache.org/jira/browse/KAFKA-129 introduced a bug in the load balancing logic of the Producer using broker.list.
      Since the broker.list doesn't specify the number of partitions in total, it should ideally pick a broker randomly, and then send the produce request with partition id -1, so that the EventHandler routes the request to a random partition.
      Instead of that, it defaults to 1 partition on each broker and ends up using the Partitioner to pick a partitions amongst the available ones.

      1. KAFKA-161.patch
        8 kB
        Neha Narkhede
      2. KAFKA-161.patch
        6 kB
        Neha Narkhede

        Activity

        Hide
        nehanarkhede Neha Narkhede added a comment -

        Fixing the load balancing strategy for the broker.list option on the Producer. With this change, when broker.list is used, the producer will pick a random broker and send request to partition -1 on that broker.

        Show
        nehanarkhede Neha Narkhede added a comment - Fixing the load balancing strategy for the broker.list option on the Producer. With this change, when broker.list is used, the producer will pick a random broker and send request to partition -1 on that broker.
        Hide
        junrao Jun Rao added a comment -

        1. We should guard logger.debug with the isDebugEnabled check.
        2. The method name getNumPartitionsForTopic is misleading since the return value is a list of partitions, instead of number of partitions. Can we rename it to sth like getPartitionListForTopic? Ditto for the name of the variable being assigned to.

        Show
        junrao Jun Rao added a comment - 1. We should guard logger.debug with the isDebugEnabled check. 2. The method name getNumPartitionsForTopic is misleading since the return value is a list of partitions, instead of number of partitions. Can we rename it to sth like getPartitionListForTopic? Ditto for the name of the variable being assigned to.
        Hide
        nehanarkhede Neha Narkhede added a comment -

        1. Guarded the logger.debug
        2. Changed the names of variables and functions to make more sense

        Show
        nehanarkhede Neha Narkhede added a comment - 1. Guarded the logger.debug 2. Changed the names of variables and functions to make more sense
        Hide
        junrao Jun Rao added a comment -

        +1 on the new patch.

        Show
        junrao Jun Rao added a comment - +1 on the new patch.

          People

          • Assignee:
            nehanarkhede Neha Narkhede
            Reporter:
            nehanarkhede Neha Narkhede
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development