[FLINK-23392] Benchmarks should exclude startup/shutdown time

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: Benchmarks
• Labels: pull-request-available
• Assignee: Chesnay Schepler
• Reporter: Chesnay Schepler
• Created: 15/Jul/21 06:18
• Updated: 21/Jul/21 15:35
• Resolved: 20/Jul/21 08:01

Description

The benchmarks currently simply run against a local environment, which means that the benchmark time includes how long it takes for Flink to start and shut down.
This creates noise in the benchmark results, and can result in serious performance regressions being reported that are not not actually performance related.

Instead, the benchmarks should setup separate setup a MiniCluster and run the job against that.

Issue Links

• blocks: FLINK-18783 Load AkkaRpcService through separate class loader
• links to: GitHub Pull Request #22