Kafka
  1. Kafka
  2. KAFKA-959

DefaultEventHandler can send more produce requests than necesary

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: core
    • Labels:
      None

      Description

      In DefaultEventHandler, for a batch of messages, it picks a random partition per message (when there is no key specified). This means that it can send up to P produce requests where P is the number of partitions in a topic. A better way is probably to pick a single random partition for the whole batch of messages. This will reduce the number of produce requests.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        21d 7h 24m 1 Guozhang Wang 25/Jul/13 00:41
        Patch Available Patch Available Resolved Resolved
        17h 26m 1 Jun Rao 25/Jul/13 18:08
        Tony Stevenson made changes -
        Workflow Apache Kafka Workflow [ 13052612 ] no-reopen-closed, patch-avail [ 13054894 ]
        Tony Stevenson made changes -
        Workflow no-reopen-closed, patch-avail [ 12796402 ] Apache Kafka Workflow [ 13052612 ]
        Jun Rao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.8 [ 12317244 ]
        Resolution Fixed [ 1 ]
        Hide
        Jun Rao added a comment -

        Thanks for the patch. +1 Committed to 0.8.

        Show
        Jun Rao added a comment - Thanks for the patch. +1 Committed to 0.8.
        Guozhang Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Guozhang Wang made changes -
        Attachment KAFKA-959.v1.patch [ 12594060 ]
        Hide
        Guozhang Wang added a comment -

        Passed unit tests

        Show
        Guozhang Wang added a comment - Passed unit tests
        Guozhang Wang made changes -
        Field Original Value New Value
        Assignee Guozhang Wang [ guozhang ]
        Hide
        Guozhang Wang added a comment -

        Proposed code change:

        1. Add a Map<Topic, PartitionIndex> to DefaultEventHandller.

        2. At the start of handle(events: Seq[KeyedMessage[K,V]]) function, clear the map.

        3. In getPartition function, if key == null, check if there is already an entry in the map with the given topic:

        3.1. If not, choose one partition from the availablePartitions, and add that partition to the map, return this partition index.

        3.2. If yes, check if its leaderBrokerIdOpt is defined; if yes, return this partition index directly, otherwise throw an exception.

        Show
        Guozhang Wang added a comment - Proposed code change: 1. Add a Map<Topic, PartitionIndex> to DefaultEventHandller. 2. At the start of handle(events: Seq[KeyedMessage [K,V] ]) function, clear the map. 3. In getPartition function, if key == null, check if there is already an entry in the map with the given topic: 3.1. If not, choose one partition from the availablePartitions, and add that partition to the map, return this partition index. 3.2. If yes, check if its leaderBrokerIdOpt is defined; if yes, return this partition index directly, otherwise throw an exception.
        Jun Rao created issue -

          People

          • Assignee:
            Guozhang Wang
            Reporter:
            Jun Rao
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development