Kafka
  1. Kafka
  2. KAFKA-959

DefaultEventHandler can send more produce requests than necesary

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: core
    • Labels:
      None

      Description

      In DefaultEventHandler, for a batch of messages, it picks a random partition per message (when there is no key specified). This means that it can send up to P produce requests where P is the number of partitions in a topic. A better way is probably to pick a single random partition for the whole batch of messages. This will reduce the number of produce requests.

        Activity

        Jun Rao created issue -
        Hide
        Guozhang Wang added a comment -

        Proposed code change:

        1. Add a Map<Topic, PartitionIndex> to DefaultEventHandller.

        2. At the start of handle(events: Seq[KeyedMessage[K,V]]) function, clear the map.

        3. In getPartition function, if key == null, check if there is already an entry in the map with the given topic:

        3.1. If not, choose one partition from the availablePartitions, and add that partition to the map, return this partition index.

        3.2. If yes, check if its leaderBrokerIdOpt is defined; if yes, return this partition index directly, otherwise throw an exception.

        Show
        Guozhang Wang added a comment - Proposed code change: 1. Add a Map<Topic, PartitionIndex> to DefaultEventHandller. 2. At the start of handle(events: Seq[KeyedMessage [K,V] ]) function, clear the map. 3. In getPartition function, if key == null, check if there is already an entry in the map with the given topic: 3.1. If not, choose one partition from the availablePartitions, and add that partition to the map, return this partition index. 3.2. If yes, check if its leaderBrokerIdOpt is defined; if yes, return this partition index directly, otherwise throw an exception.
        Guozhang Wang made changes -
        Field Original Value New Value
        Assignee Guozhang Wang [ guozhang ]
        Hide
        Guozhang Wang added a comment -

        Passed unit tests

        Show
        Guozhang Wang added a comment - Passed unit tests
        Guozhang Wang made changes -
        Attachment KAFKA-959.v1.patch [ 12594060 ]
        Guozhang Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Jun Rao added a comment -

        Thanks for the patch. +1 Committed to 0.8.

        Show
        Jun Rao added a comment - Thanks for the patch. +1 Committed to 0.8.
        Jun Rao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.8 [ 12317244 ]
        Resolution Fixed [ 1 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        21d 7h 24m 1 Guozhang Wang 25/Jul/13 00:41
        Patch Available Patch Available Resolved Resolved
        17h 26m 1 Jun Rao 25/Jul/13 18:08

          People

          • Assignee:
            Guozhang Wang
            Reporter:
            Jun Rao
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development