[FLINK-28137] Introduce SpeculativeScheduler

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.16.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Zhu Zhu
• Reporter: Zhu Zhu
• Created: 20/Jun/22 11:42
• Updated: 13/Jul/22 15:15
• Resolved: 13/Jul/22 15:15

Description

A SpeculativeScheduler will be used if speculative execution is enabled. It extends AdaptiveBatchScheduler so that speculative execution can work along with the feature to adaptively tuning parallelisms for batch jobs.

The major differences of SpeculativeScheduler are:

• SpeculativeScheduler needs to be able to directly deploy an Execution, while AdaptiveBatchScheduler can only perform ExecutionVertex level deployment.
• SpeculativeScheduler does not restart the ExecutionVertex if an execution fails when any other current execution is still making progress
• SpeculativeScheduler listens on slow tasks. Once there are slow tasks, it will block the slow nodes and deploy speculative executions of the slow tasks on other nodes.
• Once any execution finishes, SpeculativeScheduler will cancel all the remaining executions of the same execution vertex.

Issue Links

• requires: FLINK-28143 Introduce BlocklistHandler
• links to: GitHub Pull Request #20222