[FLINK-25524] If enabled changelog, RocksDB incremental checkpoint would always be full

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: Runtime / Checkpointing, Runtime / State Backends
• Labels: pull-request-available
• Assignee: Roman Khachatryan
• Reporter: Yun Tang
• Created: 05/Jan/22 08:38
• Updated: 25/Jan/22 17:11
• Resolved: 25/Jan/22 07:03

Description

Once changelog is enabled, RocksDB incremental checkpoint would only be executed during materialization. During this phase, it will leverage the materization id as the checkpoint id for RocksDB state backend's snapshot method.

However, current incremental checkpoint mechanism heavily depends on the checkpoint id. And SortedMap<Long, Set<StateHandleID>> uploadedStateIDs with checkpoint id as the key within RocksIncrementalSnapshotStrategy is the kernel for incremental checkpoint. Once we notify checkpoint complete of previous checkpoint, it will then remove the uploaded stateIds of that checkpoint, leading to we cannot get proper checkpoint information on the next RocksDBKeyedStateBackend#snapshot. That is to say, we will always upload all RocksDB artifacts.

Issue Links

• blocks: FLINK-21352 FLIP-158: Generalized incremental checkpoints
• is related to: FLINK-25144 Test Changelog State Backend manually
• links to: GitHub Pull Request #18382