Kafka
  1. Kafka
  2. KAFKA-959

DefaultEventHandler can send more produce requests than necesary

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: core
    • Labels:
      None

      Description

      In DefaultEventHandler, for a batch of messages, it picks a random partition per message (when there is no key specified). This means that it can send up to P produce requests where P is the number of partitions in a topic. A better way is probably to pick a single random partition for the whole batch of messages. This will reduce the number of produce requests.

        Activity

        Hide
        Guozhang Wang added a comment -

        Proposed code change:

        1. Add a Map<Topic, PartitionIndex> to DefaultEventHandller.

        2. At the start of handle(events: Seq[KeyedMessage[K,V]]) function, clear the map.

        3. In getPartition function, if key == null, check if there is already an entry in the map with the given topic:

        3.1. If not, choose one partition from the availablePartitions, and add that partition to the map, return this partition index.

        3.2. If yes, check if its leaderBrokerIdOpt is defined; if yes, return this partition index directly, otherwise throw an exception.

        Show
        Guozhang Wang added a comment - Proposed code change: 1. Add a Map<Topic, PartitionIndex> to DefaultEventHandller. 2. At the start of handle(events: Seq[KeyedMessage [K,V] ]) function, clear the map. 3. In getPartition function, if key == null, check if there is already an entry in the map with the given topic: 3.1. If not, choose one partition from the availablePartitions, and add that partition to the map, return this partition index. 3.2. If yes, check if its leaderBrokerIdOpt is defined; if yes, return this partition index directly, otherwise throw an exception.
        Hide
        Guozhang Wang added a comment -

        Passed unit tests

        Show
        Guozhang Wang added a comment - Passed unit tests
        Hide
        Jun Rao added a comment -

        Thanks for the patch. +1 Committed to 0.8.

        Show
        Jun Rao added a comment - Thanks for the patch. +1 Committed to 0.8.

          People

          • Assignee:
            Guozhang Wang
            Reporter:
            Jun Rao
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development