Details

    • Type: Sub-task Sub-task
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      We have 3 useful tools to run producer and consumer perf tests -

      ProducerPerformance.scala, SimpleConsumerPerformance.scala and ConsumerPerformance.scala.

      These tests expose several options that allows you to define the load for each perf run. It will be good to expose some helper scripts that will cover some single node perf testing scenarios.

      1. kafka-175-updated.patch
        26 kB
        Neha Narkhede
      2. kafka-175.patch
        36 kB
        Neha Narkhede

        Issue Links

          Activity

          Neha Narkhede created issue -
          Neha Narkhede made changes -
          Field Original Value New Value
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Neha Narkhede added a comment -

          This is a draft version for the helper scripts. By no means, is this a full set of single node perf scenarios. My feeling is that the list will evolve. We can start with some basic wrapper scripts and add more scenarios as other JIRAs.

          The following scripts are included in the patch -

          1. Plot producer throughput over batch size
          2. Plot producer throughput over time
          3. Plot simple and zk consumer throughput over fetch size

          Keep in mind that this JIRA depends on KAFKA-176 and hence is an incremental patch on top of it.

          Show
          Neha Narkhede added a comment - This is a draft version for the helper scripts. By no means, is this a full set of single node perf scenarios. My feeling is that the list will evolve. We can start with some basic wrapper scripts and add more scenarios as other JIRAs. The following scripts are included in the patch - 1. Plot producer throughput over batch size 2. Plot producer throughput over time 3. Plot simple and zk consumer throughput over fetch size Keep in mind that this JIRA depends on KAFKA-176 and hence is an incremental patch on top of it.
          Neha Narkhede made changes -
          Attachment kafka-175.patch [ 12501005 ]
          Neha Narkhede made changes -
          Link This issue depends on KAFKA-176 [ KAFKA-176 ]
          Hide
          Neha Narkhede added a comment -

          Since the scripts need working perf tools

          Show
          Neha Narkhede added a comment - Since the scripts need working perf tools
          Hide
          Jun Rao added a comment -

          A few comments:
          1. Testing on batch size is a bit confusing since how big a batch is really depends on message size. What really matters is the size of each ProduceRequest. So, maybe we should choose the request size as the variable instead.
          2. Can we merge producer-throughput-over-batch-size.sh and zk-producer-throughput-over-batch-size.sh into a single script and add an option to allow the caller to choose either broker list or zk connect?
          3. Not sure how producer-throughput-over-data-accumulated.sh is different from producer-throughput-over-batch-size.sh. They both seem to vary batch sizes.

          Show
          Jun Rao added a comment - A few comments: 1. Testing on batch size is a bit confusing since how big a batch is really depends on message size. What really matters is the size of each ProduceRequest. So, maybe we should choose the request size as the variable instead. 2. Can we merge producer-throughput-over-batch-size.sh and zk-producer-throughput-over-batch-size.sh into a single script and add an option to allow the caller to choose either broker list or zk connect? 3. Not sure how producer-throughput-over-data-accumulated.sh is different from producer-throughput-over-batch-size.sh. They both seem to vary batch sizes.
          Hide
          Neha Narkhede added a comment -

          The previous patch had some malformed lines. This patch should apply cleanly. Note to self: Switch to git !

          Show
          Neha Narkhede added a comment - The previous patch had some malformed lines. This patch should apply cleanly. Note to self: Switch to git !
          Neha Narkhede made changes -
          Attachment kafka-175-updated.patch [ 12502678 ]
          Hide
          Jun Rao added a comment -

          Previous comments 1 and 3 are not addressed.

          4. consumer-throughput-over-fetch-size.sh calls kafka-simple-consumer-perf-test.sh inconsistently, some using zk.connect and some using broker.list, some with fetch-size and some without. Also, instead of duplicating cals tol kafka-simple-consumer-perf-test.sh, can we use a for loop instead? Ditto for all other tests.

          Show
          Jun Rao added a comment - Previous comments 1 and 3 are not addressed. 4. consumer-throughput-over-fetch-size.sh calls kafka-simple-consumer-perf-test.sh inconsistently, some using zk.connect and some using broker.list, some with fetch-size and some without. Also, instead of duplicating cals tol kafka-simple-consumer-perf-test.sh, can we use a for loop instead? Ditto for all other tests.
          Gavin made changes -
          Link This issue depends on KAFKA-176 [ KAFKA-176 ]
          Gavin made changes -
          Link This issue depends upon KAFKA-176 [ KAFKA-176 ]
          Joe Stein made changes -
          Fix Version/s 0.8.1 [ 12322960 ]
          Fix Version/s 0.8 [ 12317244 ]
          Neha Narkhede made changes -
          Fix Version/s 0.9.0 [ 12323928 ]
          Fix Version/s 0.8.1 [ 12322960 ]

            People

            • Assignee:
              Neha Narkhede
              Reporter:
              Neha Narkhede
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development