[FLINK-24938] Checkpoint cleaner is closed before checkpoints are discarded

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.14.0, 1.15.0
• Fix Version/s: 1.14.3, 1.15.0
• Component/s: Runtime / Checkpointing
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Roman Khachatryan
• Created: 17/Nov/21 09:47
• Updated: 15/Dec/21 01:44
• Resolved: 23/Nov/21 12:27

Description

When CompletedCheckpointStore shuts down it tries to discard some checkpoints using CheckpointCleaner. The latter is closed asynchronously since FLINK-23647 and before the Store.

Visible as warning when running ResumeCheckpointManuallyITCase.testExternalizedIncrementalRocksDBCheckpointsZookeeper:

2021-11-17 10:47:10,599 Fail to remove checkpoint during shutdown. [DefaultCompletedCheckpointStore flink-akka.actor.default-dispatcher-5]
 java.lang.IllegalStateException: CheckpointsCleaner has already been closed
         at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193) ~[classes/:?]
         at org.apache.flink.runtime.checkpoint.CheckpointsCleaner.incrementNumberOfCheckpointsToClean(CheckpointsCleaner.java:105) ~[classes/:?]
         at org.apache.flink.runtime.checkpoint.CheckpointsCleaner.cleanup(CheckpointsCleaner.java:87) ~[classes/:?]
         at org.apache.flink.runtime.checkpoint.CheckpointsCleaner.cleanCheckpoint(CheckpointsCleaner.java:62) ~[classes/:?]
         at org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore.tryRemoveCompletedCheckpoint(DefaultCompletedCheckpointStore.java:2
         at org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore.shutdown(DefaultCompletedCheckpointStore.java:172) ~[classes/:?]
         at org.apache.flink.runtime.scheduler.SchedulerBase.shutDownCheckpointServices(SchedulerBase.java:222) ~[classes/:?]

But the test still passes.

cc: pnowojski

Issue Links

• duplicates: FLINK-24789 IllegalStateException with CheckpointCleaner being closed already
• is caused by: FLINK-23647 UnalignedCheckpointStressITCase crashed on azure
• links to: GitHub Pull Request #17886