[FLINK-13060] FailoverStrategies should respect restart constraints

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.9.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available

Release Note

Users that have enabled the "region" failover strategy, along with a restart strategy that enforces a certain number of restarts or introduces a restart delay, will see changes in behavior. This failover strategy now respects constraints that are defined by the restart strategy.

• Assignee: Chesnay Schepler
• Reporter: Chesnay Schepler
• Created: 02/Jul/19 12:29
• Updated: 28/Jul/19 18:14
• Resolved: 05/Jul/19 08:22

Description

RestartStrategies can define their own restrictions for whether job can be restarted or not. For example, they could count the number of total failures or observe failure rates.

FailoverStrategies are used for partial restarts of jobs, and currently largely bypass the restrictions defined by the restart strategies.

My proposal is the following:

Introduce a new method into the RestartStrategy interface to notify the strategy of failed task executions. Currently, strategies implicitly handle this in RestartStrategy#restart, as such the migration of our existing strategies should be trivial.

Next, before calling RestartStrategy#restart, inform the strategy about the task failure. This retains existing behavior.
Additionally, the FailoverStrategy implementation may additionally inform the restart strategy about task failures, if and when they perform a local failover. Additionally, all implementation have to check RestartStrategy#canRestart before attempting a failover.

Issue Links

• causes: FLINK-13452 Pipelined region failover strategy does not recover Job if checkpoint cannot be read
• links to: GitHub Pull Request #8972