Kafka
  1. Kafka
  2. KAFKA-88

Producer perf test fails against localhost with > 10 threads

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.6
    • Fix Version/s: 0.7
    • Component/s: config
    • Labels:
      None

      Description

      The perf test starts producing errors when it is run with --threads 11 (or higher). The cause is that we create a zookeeper connection per thread, and zookeeper recently added a feature which limits the number of connections per ip in ZOOKEEPER-336. This setting is set to 10 by default. I recommend we bump this up in our packaged zk config, since it is hard to figure this out and makes it look like the client itself is having issues.

      1. zk-config.diff
        0.4 kB
        Jay Kreps

        Activity

        Jay Kreps created issue -
        Jay Kreps made changes -
        Field Original Value New Value
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Jay Kreps added a comment -

        Change zk config to allow unlimited connections.

        Show
        Jay Kreps added a comment - Change zk config to allow unlimited connections.
        Jay Kreps made changes -
        Attachment zk-config.diff [ 12489715 ]
        Hide
        Jun Rao added a comment -

        Thanks, Jay. Looks good. Just committed this.

        Show
        Jun Rao added a comment - Thanks, Jay. Looks good. Just committed this.
        Jay Kreps made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Chris Burroughs added a comment -

        Is this a zk conn per thead only for perf code, Or do the producers/consumers do this?

        Show
        Chris Burroughs added a comment - Is this a zk conn per thead only for perf code, Or do the producers/consumers do this?
        Hide
        Jay Kreps added a comment -

        There is effectively one producer per test thread, this is good as it is more realistic i think.

        Show
        Jay Kreps added a comment - There is effectively one producer per test thread, this is good as it is more realistic i think.

          People

          • Assignee:
            Jay Kreps
            Reporter:
            Jay Kreps
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development