[FLINK-20427] Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to data loss

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.14.0
• Component/s: API / DataStream, Runtime / Checkpointing
• Labels: auto-deprioritized-critical pull-request-available

Release Note

Remove deprecated CheckpointConfig#setPreferCheckpointForRecovery

Deprecated method CheckpointConfig#setPreferCheckpointForRecovery was removed, because preferring older checkpoints over newer savepoints for recovery can lead to data loss.

• Assignee: Nicolaus Weidner
• Reporter: Till Rohrmann
• Created: 30/Nov/20 16:30
• Updated: 03/Sep/21 09:12
• Resolved: 03/Sep/21 09:12

Description

The CheckpointConfig.setPreferCheckpointForRecovery allows to configure whether Flink prefers checkpoints for recovery if the CompletedCheckpointStore contains savepoints and checkpoints. This is problematic because due to this feature, Flink might prefer older checkpoints over newer savepoints for recovery. Since some components expect that the always the latest checkpoint/savepoint is used (e.g. the SourceCoordinator), it breaks assumptions and can lead to SourceSplits which are not read. This effectively means that the system loses data. Similarly, this behaviour can cause that exactly once sinks might output results multiple times which violates the processing guarantees. Hence, I believe that we should remove this setting because it changes Flink's behaviour in some very significant way potentially w/o the user noticing.

Issue Links

• is blocked by: FLINK-20441 Deprecate CheckpointConfig.setPreferCheckpointForRecovery
• is caused by: FLINK-11159 Allow configuration whether to fall back to savepoints for restore
• is related to: FLINK-20290 Duplicated output in FileSource continuous ITCase with TM failover
• links to: GitHub Pull Request #17114