[FLINK-24344] Handling of IOExceptions when triggering checkpoints doesn't cause job failover

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.14.3, 1.15.0
• Component/s: Runtime / Checkpointing
• Labels: pull-request-available
• Assignee: Anton Kalashnikov
• Reporter: Piotr Nowojski
• Created: 21/Sep/21 08:38
• Updated: 15/Dec/21 01:44
• Resolved: 23/Sep/21 12:10

Description

When running CheckpointCoordinatorTest#testTriggerCheckpointAfterIOException, it simulates an IOException thrown from CheckpointCoordinator#initializeCheckpoint. Which makes sense but:

1. This exception never reaches CheckpointFailureManager because of the logic in CheckpointCoordinator#onTriggerFailure(). So the main purpose of the FLINK-23189 ticket is not working as intended.
2. it would be however much much better to throw IOException from checkpointStorageView.initializeLocationForCheckpoint rather from checkpointIdCounter.getAndIncrement. The latter could be refactored out, while the production IOException can be thrown in reality from initializeLocationForCheckpoint)

Issue Links

• relates to: FLINK-24249 login from keytab fail when disk damage
• relates to: FLINK-23189 Count and fail the task when the disk is error on JobManager
• links to: GitHub Pull Request #17331