Qpid
  1. Qpid
  2. QPID-4089

Add latency tests into Java Performance Tests

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.17
    • Fix Version/s: 0.17
    • Component/s: Java Performance Tests
    • Labels:
      None

      Description

      We need to implement a group of tests measuring latency metrics (minimum latency, maximum latency, average latency, standard deviation) by varying the following attributes:
      queue type
      message size
      number of producers and consumers
      selectors

      The publishing rate should be deliberately limited in such tests to never throttle the broker or exceed the consumption rate.
      Ideally, a special configuration parameter like expectedPublishingRate can be added to control publishing, however, we can use currently implemented "interval" parameter. We can also publish data in small batches with delays between batches.

      Also, it would be interesting to see comparison of latency vs throughput by varying the publishing rate.

      Publishing in AA mode is only required for initial latency tests.

        Activity

        Hide
        Alex Rudyy added a comment -

        Patch adding functionality to measure latency into consumer client.

        Show
        Alex Rudyy added a comment - Patch adding functionality to measure latency into consumer client.
        Hide
        Alex Rudyy added a comment -

        Keith,

        Could you please review and commit the patch adding latency tests?

        Show
        Alex Rudyy added a comment - Keith, Could you please review and commit the patch adding latency tests?
        Hide
        Keith Wall added a comment -

        Hi Alex

        A couple of comments:

        1) AggregatedSeriesResults#addMessageLatency appears to be unused.

        2) The name AggregatedSeriesResults does not clearly identify the purpose of the class. We've already used the word 'aggregation' elsewhere to mean the process of summarising many ParticipantResults into a single ParticipantResult. Since this is not the concern of this class, I'd use a different word.

        3) ConsumerParticipantLatencyResult#ConsumerParticipantLatencyResult(ConsumerParticipantResult result) is fragile. If a maintainer changes ConsumerParticipantResult they now need to change this class too. I'd prefer we avoid this.

        4) ConsumerParticipantLatencyResult typo Lateny

        5) ConsumerParticipantResultAggregator#getLatencyAggregatedResults ununsed

        6) ConsumerParticipantResultAggregator#getAggregatedResult(). I think this method surprises the reader by returning a ConsumerParticipantLatencyResult. It is also fragile as we have repeated implementation (calls to setRolledUpConstantAttributes and setComputedVariableAttributes) from the base class.

        7) In some of the PNGs the x-labels are being ellipsis (...)

        8) It would be nice if we could make use of error bars to show the stdev, rather than the separate column (one for the future i think) http://www.jfree.org/jfreechart/api/javadoc/org/jfree/chart/renderer/category/StatisticalBarRenderer.html

        9) Suggestion: In the chartdef files, you could make use of chartSubtitle to avoid some of the very long chart titles.

        Show
        Keith Wall added a comment - Hi Alex A couple of comments: 1) AggregatedSeriesResults#addMessageLatency appears to be unused. 2) The name AggregatedSeriesResults does not clearly identify the purpose of the class. We've already used the word 'aggregation' elsewhere to mean the process of summarising many ParticipantResults into a single ParticipantResult. Since this is not the concern of this class, I'd use a different word. 3) ConsumerParticipantLatencyResult#ConsumerParticipantLatencyResult(ConsumerParticipantResult result) is fragile. If a maintainer changes ConsumerParticipantResult they now need to change this class too. I'd prefer we avoid this. 4) ConsumerParticipantLatencyResult typo Lateny 5) ConsumerParticipantResultAggregator#getLatencyAggregatedResults ununsed 6) ConsumerParticipantResultAggregator#getAggregatedResult(). I think this method surprises the reader by returning a ConsumerParticipantLatencyResult. It is also fragile as we have repeated implementation (calls to setRolledUpConstantAttributes and setComputedVariableAttributes) from the base class. 7) In some of the PNGs the x-labels are being ellipsis (...) 8) It would be nice if we could make use of error bars to show the stdev, rather than the separate column (one for the future i think) http://www.jfree.org/jfreechart/api/javadoc/org/jfree/chart/renderer/category/StatisticalBarRenderer.html 9) Suggestion: In the chartdef files, you could make use of chartSubtitle to avoid some of the very long chart titles.
        Hide
        Keith Wall added a comment -

        Hi Alex

        Changes look good, but I see one failing test:

        Testcase: testAggregateResultsForConsumerWithLatencyResults took 0.002 sec
                FAILED
        Unexpected average expected:<5.0> but was:<0.0>
        junit.framework.AssertionFailedError: Unexpected average expected:<5.0> but was:<0.0>
                at org.apache.qpid.disttest.results.aggregation.TestResultAggregatorTest.assertLatencyAggregatedResults(TestResultAggregatorTest.java:167)
                at org.apache.qpid.disttest.results.aggregation.TestResultAggregatorTest.testAggregateResultsForConsumerWithLatencyResults(TestResultAggregatorTest.java:160)
        
        Show
        Keith Wall added a comment - Hi Alex Changes look good, but I see one failing test: Testcase: testAggregateResultsForConsumerWithLatencyResults took 0.002 sec FAILED Unexpected average expected:<5.0> but was:<0.0> junit.framework.AssertionFailedError: Unexpected average expected:<5.0> but was:<0.0> at org.apache.qpid.disttest.results.aggregation.TestResultAggregatorTest.assertLatencyAggregatedResults(TestResultAggregatorTest.java:167) at org.apache.qpid.disttest.results.aggregation.TestResultAggregatorTest.testAggregateResultsForConsumerWithLatencyResults(TestResultAggregatorTest.java:160)
        Hide
        Alex Rudyy added a comment -

        Hi Keith,

        I am sorry I attached wrong patch with not fixed tests.

        With latest patch attached at 30/Jun/12 23:08 I sorted majority of the issues.

        Only issues left to resolve is a reporting order of "All Consumers" results. It appears as a first consumer result now instead of the last as it was before. At the moment I changed tests to pass but that needs to be changed.

        Also, I have not fixed the issue with x-labels being displayed as ellipsis (...)

        The rest of comments I tried to address in the patch attached.

        Show
        Alex Rudyy added a comment - Hi Keith, I am sorry I attached wrong patch with not fixed tests. With latest patch attached at 30/Jun/12 23:08 I sorted majority of the issues. Only issues left to resolve is a reporting order of "All Consumers" results. It appears as a first consumer result now instead of the last as it was before. At the moment I changed tests to pass but that needs to be changed. Also, I have not fixed the issue with x-labels being displayed as ellipsis (...) The rest of comments I tried to address in the patch attached.
        Hide
        Keith Wall added a comment -

        Hi Alex

        I tried the new patch but got a compile error.

         [javac] symbol  : variable SERIES
            [javac] location: class org.apache.qpid.disttest.results.aggregation.SeriesStatistics
            [javac]                 ((ConsumerParticipantResult)participantResult).setMessageLatencies(SeriesStatistics.SERIES);
            [javac]                                                                                                    ^
            [javac] Note: Some input files use unchecked or unsafe operations.
            [javac] Note: Recompile with -Xlint:unchecked for details.
            [javac] 1 error
        
        Show
        Keith Wall added a comment - Hi Alex I tried the new patch but got a compile error. [javac] symbol : variable SERIES [javac] location: class org.apache.qpid.disttest.results.aggregation.SeriesStatistics [javac] ((ConsumerParticipantResult)participantResult).setMessageLatencies(SeriesStatistics.SERIES); [javac] ^ [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 1 error
        Hide
        Alex Rudyy added a comment -

        Hi Keith,

        I fixed the compilation issue in an updated patch.

        Show
        Alex Rudyy added a comment - Hi Keith, I fixed the compilation issue in an updated patch.
        Hide
        Keith Wall added a comment -

        Patch applied.

        Show
        Keith Wall added a comment - Patch applied.

          People

          • Assignee:
            Keith Wall
            Reporter:
            Alex Rudyy
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development