[FLINK-15731] Stop while Checkpoint is In-Progress Triggers Job Failover

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 1.9.1, 1.10.0
• Fix Version/s: 1.10.0
• Component/s: Runtime / Checkpointing

Release Note

I unsuccessfully tried to reproduce the issue today. Not sure, why I was seeing this issue so consistently on Wednesday. Only explanation I can think of right now, is that the job had some unrelated issue that only manifested itself in this situation.

The original theory for the reason was wrong anyhow as it turns out aborted checkpoints due to suspension are ignored in the checkpoint failure count.

• Assignee: Unassigned
• Reporter: Konstantin Knauf
• Created: 22/Jan/20 16:25
• Updated: 24/Jan/20 15:47
• Resolved: 24/Jan/20 15:47

Description

Currently, when a Job is stopped in-progress checkpoints are aborted and afterwards a synchronous savepoint is started.

Since the number of tolerable checkpoint failures is 0 per default (see org.apache.flink.streaming.api.environment.CheckpointConfig#getTolerableCheckpointFailureNumber), this triggers a restart of the job if there are any ongoing checkpoints.

In consequence, the stop call only triggers a failover of the job instead of stopping the job, if there is an ongoing checkpoint (or savepoint).

Possible options I see are:

a) change default of tolerable checkpoint failures to at least the max number of concurrent checkpoints
b) do not count checkpoint failures due to the stop action when checking against tolerable checkpoint failures
c) do not abort pending checkpoints when stopping a job, but queue the synchronous savepoint after all current in-progress checkpoints