Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-23949

first incremental checkpoint after a savepoint will degenerate into a full checkpoint

    XMLWordPrintableJSON

Details

    Description

      In RocksIncrementalSnapshotStrategy we will record the uploaded rocksdb files corresponding to the checkpoint id,and clean it in CheckpointListener#notifyCheckpointComplete .

      @Override
      public void notifyCheckpointComplete(long completedCheckpointId) {
          synchronized (materializedSstFiles) {
              if (completedCheckpointId > lastCompletedCheckpointId) {
                  materializedSstFiles
                          .keySet()
                          .removeIf(checkpointId -> checkpointId < completedCheckpointId);
                  lastCompletedCheckpointId = completedCheckpointId;
              }
          }
      }

       

      This works well without savepoint, but when a savepoint is completed, it will clean up the materializedSstFiles of the previous checkpoint. It leads to the first checkpoint after the savepoint must upload all files in rocksdb.

      Solving the problem is also very simple, I propose to clean materializedSstFiles and update  lastCompletedCheckpointId only when materializedSstFiles.keySet().contains(completedCheckpointId) .

      If a completedCheckpointId is not in materializedSstFiles.keySet() , there are only two cases:

      1. It is a checkpoint but there is a checkpoint with larger id number completed before it

      2. It is a savepoint (savepoint not produce by RocksIncrementalSnapshotStrategy)

      In either case we don’t need clean materializedSstFiles and update  lastCompletedCheckpointId  anymore.

      yunta , trohrmann , I have submitted a pull request to solve this problem, please evaluate whether it is appropriate.

       

      Attachments

        Activity

          People

            Feifan Wang Feifan Wang
            Feifan Wang Feifan Wang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: