[FLINK-10751] Checkpoints should be retained when job reaches suspended state

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 1.6.2, 1.7.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Ufuk Celebi
• Reporter: Ufuk Celebi
• Created: 01/Nov/18 14:31
• Updated: 11/Dec/18 09:31
• Resolved: 11/Dec/18 09:30

Description

CheckpointProperties define in which terminal job status a checkpoint should be disposed.

I've noticed that the properties for CHECKPOINT_NEVER_RETAINED, CHECKPOINT_RETAINED_ON_FAILURE prescribe checkpoint disposal in (locally) terminal job status SUSPENDED.

Since a job reaches the SUSPENDED state when its JobMaster looses leadership, this would result in the checkpoint to be cleaned up and not being available for recovery by the new leader. Therefore, we should rather retain checkpoints when reaching job status SUSPENDED.

BUT: Because we special case this terminal state in the only highly available CompletedCheckpointStore implementation (see ZooKeeperCompletedCheckpointStore) and don't use regular checkpoint disposal, this issue has not surfaced yet.

I think we should proactively fix the properties to indicate to retain checkpoints in SUSPENDED state. We might actually completely remove this case since with this change, all properties will indicate to retain on suspension.

Issue Links

• links to: GitHub Pull Request #7006