Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-543

Metadata request from DefaultEventHandler.handle repeats same topic over and over

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.8.0
    • None
    • None

    Description

      It looks like we are calling BrokerPartitionInfo.updateInfo() with a list of the same topic repeated many times:

      Here is the line:
      Utils.swallowError(brokerPartitionInfo.updateInfo(outstandingProduceRequests.map(_.getTopic)))

      The outstandingProduceRequests can (and generally would) have many entries for the same topic.

      For example if I use the producer performance test with the default batch size on a topic "test" my metadata request will have the topic "test" repeated 200 times. On the server side we do several zk reads for each of these repetitions.

      This is causing the metadata api to timeout in my perf test periodically.

      I think the fix is simply to de-duplicate prior to the call (and perhaps again on the server in case of a misbehaving client).

      Attachments

        1. KAFKA-543.patch
          2 kB
          Jay Kreps
        2. KAFKA-543-v2.patch
          21 kB
          Jay Kreps

        Activity

          People

            jkreps Jay Kreps
            jkreps Jay Kreps
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: