Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.11.4, 1.12.5, 1.13.2
Description
In RocksIncrementalSnapshotStrategy we will record the uploaded rocksdb files corresponding to the checkpoint id，and clean it in CheckpointListener#notifyCheckpointComplete .
@Override public void notifyCheckpointComplete(long completedCheckpointId) { synchronized (materializedSstFiles) { if (completedCheckpointId > lastCompletedCheckpointId) { materializedSstFiles .keySet() .removeIf(checkpointId -> checkpointId < completedCheckpointId); lastCompletedCheckpointId = completedCheckpointId; } } }
This works well without savepoint, but when a savepoint is completed, it will clean up the materializedSstFiles of the previous checkpoint. It leads to the first checkpoint after the savepoint must upload all files in rocksdb.
Solving the problem is also very simple, I propose to clean materializedSstFiles and update lastCompletedCheckpointId only when materializedSstFiles.keySet().contains(completedCheckpointId) .
If a completedCheckpointId is not in materializedSstFiles.keySet() , there are only two cases：
1. It is a checkpoint but there is a checkpoint with larger id number completed before it
2. It is a savepoint (savepoint not produce by RocksIncrementalSnapshotStrategy)
In either case we don’t need clean materializedSstFiles and update lastCompletedCheckpointId anymore.
yunta , trohrmann , I have submitted a pull request to solve this problem, please evaluate whether it is appropriate.
