[FLINK-29825] Improve benchmark stability

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.17.0
• Component/s: Benchmarks
• Labels: pull-request-available
• Assignee: Yanfei Lei
• Reporter: Yanfei Lei
• Created: 01/Nov/22 04:35
• Updated: 22/Mar/23 11:31
• Resolved: 10/Mar/23 07:36

Description

Currently, regressions are detected by a simple script which may have false positives and false negatives, especially for benchmarks with small absolute values, small value changes would cause large percentage changes. see here for details.
And all benchmarks are executed on one physical machine, it might happen that hardware issues affect performance, like "FLINK-18614 Performance regression 2020.07.13".
 
This ticket aims to improve the precision and recall of the regression-check script.
 

Issue Links

• causes: FLINK-31561 flink-master-regression-check is failing since March 15, 2023
• is a parent of: FLINK-27571 Recognize "less is better" benchmarks in regression detection script
• links to: GitHub Pull Request #66

Sub-Tasks

• Backup codespeed‘s data regularly: Open