Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20427

Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to data loss

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Hide
      Remove deprecated CheckpointConfig#setPreferCheckpointForRecovery

      Deprecated method CheckpointConfig#setPreferCheckpointForRecovery was removed, because preferring older checkpoints over newer savepoints for recovery can lead to data loss.
      Show
      Remove deprecated CheckpointConfig#setPreferCheckpointForRecovery Deprecated method CheckpointConfig#setPreferCheckpointForRecovery was removed, because preferring older checkpoints over newer savepoints for recovery can lead to data loss.

    Description

      The CheckpointConfig.setPreferCheckpointForRecovery allows to configure whether Flink prefers checkpoints for recovery if the CompletedCheckpointStore contains savepoints and checkpoints. This is problematic because due to this feature, Flink might prefer older checkpoints over newer savepoints for recovery. Since some components expect that the always the latest checkpoint/savepoint is used (e.g. the SourceCoordinator), it breaks assumptions and can lead to SourceSplits which are not read. This effectively means that the system loses data. Similarly, this behaviour can cause that exactly once sinks might output results multiple times which violates the processing guarantees. Hence, I believe that we should remove this setting because it changes Flink's behaviour in some very significant way potentially w/o the user noticing.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Nicolaus Weidner Nicolaus Weidner
            trohrmann Till Rohrmann
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment