Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-10091

Benchmark instability in RedisZaddAndZremBenchmark

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 1.15.0
    • None
    • benchmarks, redis

    Description

      This is ITERATION 1 of benchmarking against baseline.
                                                 RedisGetBenchmark avg ops/sec  Baseline:    428358.55  Test:    446241.49  Difference:   +4.2%
                                                                   avg latency  Baseline:   1678809.55  Test:   1611045.34  Difference:   -4.0%
                                                RedisHgetBenchmark avg ops/sec  Baseline:    438535.25  Test:    441078.97  Difference:   +0.6%
                                                                   avg latency  Baseline:   1638968.92  Test:   1630999.16  Difference:   -0.5%
                                         RedisHsetAndHgetBenchmark avg ops/sec  Baseline:    191331.54  Test:    186806.37  Difference:   -2.4%
                                                                   avg latency  Baseline:   3759694.09  Test:   3850396.04  Difference:   +2.4%
                                                RedisHsetBenchmark avg ops/sec  Baseline:    308079.41  Test:    303715.78  Difference:   -1.4%
                                                                   avg latency  Baseline:   2332102.91  Test:   2367639.93  Difference:   +1.5%
                                                 RedisSetBenchmark avg ops/sec  Baseline:    318147.74  Test:    326996.22  Difference:   +2.8%
                                                                   avg latency  Baseline:   2263687.57  Test:   2198181.68  Difference:   -2.9%
                                 RedisWeightedHsetAndHgetBenchmark avg ops/sec  Baseline:    390383.79  Test:    385565.48  Difference:   -1.2%
                                                                   avg latency  Baseline:   1842561.96  Test:   1864716.09  Difference:   +1.2%
                               RedisWeightedZaddAndZrangeBenchmark avg ops/sec  Baseline:    375078.82  Test:    350886.03  Difference:   -6.5%
                                                                   avg latency  Baseline:   1917476.15  Test:   2049452.87  Difference:   +6.9%
                                         RedisZaddAndZremBenchmark avg ops/sec  Baseline:    124763.60  Test:    118600.72  Difference:   -4.9%
                                                                   avg latency  Baseline:   5766048.34  Test:   6065491.13  Difference:   +5.2%
                                                RedisZaddBenchmark avg ops/sec  Baseline:    432494.29  Test:    438594.53  Difference:   +1.4%
                                                                   avg latency  Baseline:   1662639.09  Test:   1639906.18  Difference:   -1.4%
                                              RedisZrangeBenchmark avg ops/sec  Baseline:    295523.77  Test:    340381.87  Difference:  +15.2%
                                                                   avg latency  Baseline:   2433250.54  Test:   2112961.66  Difference:  -13.2%
                                       RedisZrangeByScoreBenchmark avg ops/sec  Baseline:    285836.33  Test:    356142.91  Difference:  +24.6%
                                                                   avg latency  Baseline:   2515708.23  Test:   2019249.68  Difference:  -19.7%
      This is ITERATION 2 of benchmarking against baseline.
                               RedisWeightedZaddAndZrangeBenchmark avg ops/sec  Baseline:    358570.33  Test:    348012.02  Difference:   -2.9%
                                                                   avg latency  Baseline:   2005804.05  Test:   2066535.24  Difference:   +3.0%
                                         RedisZaddAndZremBenchmark avg ops/sec  Baseline:    122900.38  Test:    115226.22  Difference:   -6.2%
                                                                   avg latency  Baseline:   5858357.13  Test:   6243018.38  Difference:   +6.6%
      This is ITERATION 3 of benchmarking against baseline.
                                         RedisZaddAndZremBenchmark avg ops/sec  Baseline:    125677.36  Test:    118359.94  Difference:   -5.8%
                                                                   avg latency  Baseline:   5721859.26  Test:   6082944.28  Difference:   +6.3%
      This is ITERATION 4 of benchmarking against baseline.
                                         RedisZaddAndZremBenchmark avg ops/sec  Baseline:    122099.00  Test:    115864.88  Difference:   -5.1%
                                                                   avg latency  Baseline:   5893264.03  Test:   6203439.89  Difference:   +5.3%
      This is ITERATION 5 of benchmarking against baseline.
                                         RedisZaddAndZremBenchmark avg ops/sec  Baseline:    122234.53  Test:    116038.15  Difference:   -5.1%
                                                                   avg latency  Baseline:   5883499.99  Test:   6202493.46  Difference:   +5.4%

      ===========================================
      This benchmark run is the result of comparing b7297023c309210e70f3792980e75d10c54e7421 with baseline geode-for-redis-benchmark-baseline
      ================ Commit Message ===================
      commit b7297023c309210e70f3792980e75d10c54e7421
      Author: Jens Deppe <jdeppe@vmware.com>
      Date:   Fri Feb 25 13:41:18 2022 -0800

          GEODE-10083: Fix RedisProxy to correctly process MOVED response (#7394)
          
          - When a MOVED response is received, the already queued processor needs
            to be discarded otherwise subsequent requests may possibly be
            corrupted.
      =============== Benchmark Results URI =================
      http://files.apachegeode-ci.info/benchmarks/apache-develop-main/b7297023c309210e70f3792980e75d10c54e7421/benchmarks-apache-develop-main-benchmark-radish-171-31258406-radish.tgz

      ===========================================

       

      The above benchmark failed on all 5 iterations of this run, but passed after one repeat when the concourse job was rerun against the same commit:

      ===========================================
      This benchmark run is the result of comparing b7297023c309210e70f3792980e75d10c54e7421 with baseline geode-for-redis-benchmark-baseline
      ================ Commit Message ===================
      commit b7297023c309210e70f3792980e75d10c54e7421
      Author: Jens Deppe <jdeppe@vmware.com>
      Date:   Fri Feb 25 13:41:18 2022 -0800

          GEODE-10083: Fix RedisProxy to correctly process MOVED response (#7394)
          
          - When a MOVED response is received, the already queued processor needs
            to be discarded otherwise subsequent requests may possibly be
            corrupted.
      =============== Benchmark Results URI =================
      http://files.apachegeode-ci.info/benchmarks/apache-develop-main/b7297023c309210e70f3792980e75d10c54e7421/benchmarks-apache-develop-main-benchmark-radish-172-31385041-radish.tgz
      ===========================================

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zoernere Eric Zoerner
            donalevans Donal Evans
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment