[FLINK-30707] Let speculative execution take input data amount into account when detecting slow tasks

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.17.0
• Fix Version/s: 1.17.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available

Release Note

The slow task detecting is improved for speculative execution. Previously, it only considers the execution time of tasks when deciding which tasks are slow. It now takes the input data volume of tasks into account. Tasks which has a longer execution time but consumes more data may not be considered as slow. This improvement helps to eliminate the negative impacts of data skew on slow task detecting.

• Assignee: xingbe
• Reporter: xingbe
• Created: 17/Jan/23 03:01
• Updated: 08/Feb/23 03:46
• Resolved: 08/Feb/23 03:46

Description

The default slowTaskDetector in speculative execution is based on execution time now, if the data amount allocated to each subtask is uneven, time-based slow task detection may not reflect the actual performance of the subtask. Therefore, it is necessary to take the input information into account when detecting slow tasks.

The major changes are:

• Need to know the amount of data processed by each execution vertex.
• ExecutionTimeBasedSlowTaskDetector needs to introduce the amount of input data to weight it with execution time.

Issue Links

• links to: GitHub Pull Request #21695