Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-9340

Benchmark instability in PartitionedPutLongBenchmark

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.15.0
    • Fix Version/s: None
    • Component/s: benchmarks

      Description

      PartitionedPutLongBenchmark failed in CI (https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/benchmark-base/builds/6):

      This is ITERATION 1 of benchmarking against baseline.
                                        P2pPartitionedGetBenchmark avg ops/sec  Baseline:    825011.38  Test:    835847.67  Difference:   +1.3%
                                                                   avg latency  Baseline:    871392.31  Test:    859444.66  Difference:   -1.4%
                                        P2pPartitionedPutBenchmark avg ops/sec  Baseline:    123838.43  Test:    122686.30  Difference:   -0.9%
                                                                   avg latency  Baseline:   6015719.73  Test:   6119472.19  Difference:   +1.7%
                                   P2pPartitionedPutBytesBenchmark avg ops/sec  Baseline:    174887.77  Test:    171040.93  Difference:   -2.2%
                                                                   avg latency  Baseline:   4145337.60  Test:   4236159.60  Difference:   +2.2%
                             PartitionedFunctionExecutionBenchmark avg ops/sec  Baseline:    248635.36  Test:    261498.94  Difference:   +5.2%
                                                                   avg latency  Baseline:    867122.63  Test:    824550.34  Difference:   -4.9%
                PartitionedFunctionExecutionWithArgumentsBenchmark avg ops/sec  Baseline:    280071.19  Test:    275305.31  Difference:   -1.7%
                                                                   avg latency  Baseline:   1026643.12  Test:   1044307.43  Difference:   +1.7%
                  PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec  Baseline:    301416.23  Test:    304317.30  Difference:   +1.0%
                                                                   avg latency  Baseline:   1908390.88  Test:   1890040.46  Difference:   -1.0%
                                           PartitionedGetBenchmark avg ops/sec  Baseline:    790800.52  Test:    784514.74  Difference:   -0.8%
                                                                   avg latency  Baseline:    908357.58  Test:    915790.96  Difference:   +0.8%
                                       PartitionedGetLongBenchmark avg ops/sec  Baseline:   1020821.32  Test:    996529.93  Difference:   -2.4%
                                                                   avg latency  Baseline:    703761.09  Test:    720744.36  Difference:   +2.4%
                                     PartitionedGetStringBenchmark avg ops/sec  Baseline:   1028992.93  Test:   1010447.47  Difference:   -1.8%
                                                                   avg latency  Baseline:    698009.55  Test:    710765.29  Difference:   +1.8%
                                  PartitionedIndexedQueryBenchmark avg ops/sec  Baseline:     30868.78  Test:     31478.90  Difference:   +2.0%
                                                                   avg latency  Baseline:  18670093.21  Test:  18278083.16  Difference:   -2.1%
                               PartitionedNonIndexedQueryBenchmark avg ops/sec  Baseline:        99.45  Test:       101.97  Difference:   +2.5%
                                                                   avg latency  Baseline: 723415530.75  Test: 705653061.86  Difference:   -2.5%
                                        PartitionedPutAllBenchmark avg ops/sec  Baseline:      7921.61  Test:      7816.66  Difference:   -1.3%
                                                                   avg latency  Baseline:  18172638.37  Test:  18416169.28  Difference:   +1.3%
                                    PartitionedPutAllLongBenchmark avg ops/sec  Baseline:      1379.53  Test:      1169.16  Difference:  -15.2%
                                                                   avg latency  Baseline: 105140260.44  Test: 123722914.94  Difference:  +17.7%
                                           PartitionedPutBenchmark avg ops/sec  Baseline:    474986.11  Test:    467924.19  Difference:   -1.5%
                                                                   avg latency  Baseline:   1514276.31  Test:   1536263.99  Difference:   +1.5%
                                      PartitionedPutBytesBenchmark avg ops/sec  Baseline:    457550.69  Test:    456011.33  Difference:   -0.3%
                                                                   avg latency  Baseline:   1570713.84  Test:   1575841.02  Difference:   +0.3%
                                       PartitionedPutLongBenchmark avg ops/sec  Baseline:    418221.79  Test:    389221.70  Difference:   -6.9%
                                                                   avg latency  Baseline:   1717869.66  Test:   1849602.96  Difference:   +7.7%
                                     PartitionedPutStringBenchmark avg ops/sec  Baseline:    410007.93  Test:    390442.31  Difference:   -4.8%
                                                                   avg latency  Baseline:   1754915.95  Test:   1841243.74  Difference:   +4.9%
                              ReplicatedFunctionExecutionBenchmark avg ops/sec  Baseline:    478780.89  Test:    466855.62  Difference:   -2.5%
                                                                   avg latency  Baseline:    449616.85  Test:    461217.05  Difference:   +2.6%
                 ReplicatedFunctionExecutionWithArgumentsBenchmark avg ops/sec  Baseline:    504979.76  Test:    501025.09  Difference:   -0.8%
                                                                   avg latency  Baseline:   2279046.16  Test:   2296973.89  Difference:   +0.8%
                   ReplicatedFunctionExecutionWithFiltersBenchmark avg ops/sec  Baseline:    523232.66  Test:    517943.28  Difference:   -1.0%
                                                                   avg latency  Baseline:   1374146.10  Test:   1387969.18  Difference:   +1.0%
                                            ReplicatedGetBenchmark avg ops/sec  Baseline:    799001.77  Test:    788077.23  Difference:   -1.4%
                                                                   avg latency  Baseline:    899092.08  Test:    911932.02  Difference:   +1.4%
                                        ReplicatedGetLongBenchmark avg ops/sec  Baseline:   1038120.92  Test:   1025267.47  Difference:   -1.2%
                                                                   avg latency  Baseline:    691970.73  Test:    700641.58  Difference:   +1.3%
                                   ReplicatedIndexedQueryBenchmark avg ops/sec  Baseline:     32088.81  Test:     31641.88  Difference:   -1.4%
                                                                   avg latency  Baseline:  17960402.47  Test:  18222972.89  Difference:   +1.5%
                                ReplicatedNonIndexedQueryBenchmark avg ops/sec  Baseline:        49.31  Test:        48.15  Difference:   -2.4%
                                                                   avg latency  Baseline: 1456350672.39  Test: 1490270040.10  Difference:   +2.3%
                                         ReplicatedPutAllBenchmark avg ops/sec  Baseline:       821.20  Test:       793.66  Difference:   -3.4%
                                                                   avg latency  Baseline: 175498155.76  Test: 181712150.69  Difference:   +3.5%
                                     ReplicatedPutAllLongBenchmark avg ops/sec  Baseline:       855.75  Test:       812.41  Difference:   -5.1%
                                                                   avg latency  Baseline: 168343905.55  Test: 177510321.30  Difference:   +5.4%
                                            ReplicatedPutBenchmark avg ops/sec  Baseline:    485135.10  Test:    481879.66  Difference:   -0.7%
                                                                   avg latency  Baseline:   1482039.07  Test:   1491885.72  Difference:   +0.7%
                                        ReplicatedPutLongBenchmark avg ops/sec  Baseline:    454212.92  Test:    443628.38  Difference:   -2.3%
                                                                   avg latency  Baseline:   1583988.26  Test:   1620588.33  Difference:   +2.3%
      This is ITERATION 2 of benchmarking against baseline.
                                    PartitionedPutAllLongBenchmark avg ops/sec  Baseline:      1359.77  Test:      1263.03  Difference:   -7.1%
                                                                   avg latency  Baseline: 106472640.40  Test: 114522255.46  Difference:   +7.6%
                                       PartitionedPutLongBenchmark avg ops/sec  Baseline:    426518.85  Test:    400573.57  Difference:   -6.1%
                                                                   avg latency  Baseline:   1686679.27  Test:   1793189.27  Difference:   +6.3%
                                     ReplicatedPutAllLongBenchmark avg ops/sec  Baseline:       847.77  Test:       802.20  Difference:   -5.4%
                                                                   avg latency  Baseline: 170110197.18  Test: 179656637.86  Difference:   +5.6%
      This is ITERATION 3 of benchmarking against baseline.
                                    PartitionedPutAllLongBenchmark avg ops/sec  Baseline:      1308.38  Test:      1371.30  Difference:   +4.8%
                                                                   avg latency  Baseline: 110742328.38  Test: 105657848.59  Difference:   -4.6%
                                       PartitionedPutLongBenchmark avg ops/sec  Baseline:    422675.38  Test:    393034.80  Difference:   -7.0%
                                                                   avg latency  Baseline:   1701984.68  Test:   1830333.62  Difference:   +7.5%
                                     ReplicatedPutAllLongBenchmark avg ops/sec  Baseline:       882.11  Test:       868.74  Difference:   -1.5%
                                                                   avg latency  Baseline: 163447771.46  Test: 166032265.56  Difference:   +1.6%
      This is ITERATION 4 of benchmarking against baseline.
                                       PartitionedPutLongBenchmark avg ops/sec  Baseline:    421497.23  Test:    394141.73  Difference:   -6.5%
                                                                   avg latency  Baseline:   1705094.48  Test:   1823187.55  Difference:   +6.9%
      This is ITERATION 5 of benchmarking against baseline.
                                       PartitionedPutLongBenchmark avg ops/sec  Baseline:    410979.61  Test:    387125.91  Difference:   -5.8%
                                                                   avg latency  Baseline:   1749420.68  Test:   1854964.43  Difference:   +6.0%
      

      Potentially, there is some instability in this benchmark since it passed in the runs immediately before and after.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                balesh2 Hale Bales
                Reporter:
                sabbey37 Sarah Abbey
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: