Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-88

Producer perf test fails against localhost with > 10 threads

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.6
    • Fix Version/s: 0.7
    • Component/s: config
    • Labels:
      None

      Description

      The perf test starts producing errors when it is run with --threads 11 (or higher). The cause is that we create a zookeeper connection per thread, and zookeeper recently added a feature which limits the number of connections per ip in ZOOKEEPER-336. This setting is set to 10 by default. I recommend we bump this up in our packaged zk config, since it is hard to figure this out and makes it look like the client itself is having issues.

      1. zk-config.diff
        0.4 kB
        Jay Kreps

        Activity

        Hide
        jkreps Jay Kreps added a comment -

        Change zk config to allow unlimited connections.

        Show
        jkreps Jay Kreps added a comment - Change zk config to allow unlimited connections.
        Hide
        junrao Jun Rao added a comment -

        Thanks, Jay. Looks good. Just committed this.

        Show
        junrao Jun Rao added a comment - Thanks, Jay. Looks good. Just committed this.
        Hide
        cburroughs Chris Burroughs added a comment -

        Is this a zk conn per thead only for perf code, Or do the producers/consumers do this?

        Show
        cburroughs Chris Burroughs added a comment - Is this a zk conn per thead only for perf code, Or do the producers/consumers do this?
        Hide
        jkreps Jay Kreps added a comment -

        There is effectively one producer per test thread, this is good as it is more realistic i think.

        Show
        jkreps Jay Kreps added a comment - There is effectively one producer per test thread, this is good as it is more realistic i think.

          People

          • Assignee:
            jkreps Jay Kreps
            Reporter:
            jkreps Jay Kreps
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development