[FLINK-28843] Fail to find incremental handle when restoring from changelog checkpoint in claim mode

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.15.0, 1.15.1
• Fix Version/s: 1.16.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: Lihe Ma
• Reporter: Lihe Ma
• Created: 06/Aug/22 03:09
• Updated: 09/Aug/22 03:10
• Resolved: 09/Aug/22 03:10

Description

1. When native checkpoint is enabled and incremental checkpointing is enabled in rocksdb statebackend，if state data is greater than state.storage.fs.memory-threshold，it will be stored in a data file (FileStateHandle，RelativeFileStateHandle, etc) rather than stored with ByteStreamStateHandle in checkpoint metadata, like base-path1/chk-1/file1.
2. Then restore the job from base-path1/chk-1 in claim mode，using changelog statebackend，and the checkpoint path is set to base-path2, then new checkpoint will be saved in base-path2/chk-2, previous checkpoint file (base-path1/chk-1/file1) is needed.
3. Then restore the job from base-path2/chk-2 in changelog statebackend, flink will try to read base-path2/chk-2/file1, rather than the actual file location base-path1/chk-1/file1, which leads to FileNotFoundException and job failed.

 
How to reproduce?

1. Set state.storage.fs.memory-threshold to a small value, like '20b'.
2. run org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationSwitchStateBackendITCase#testSwitchFromDisablingToEnablingInClaimMode

Issue Links

• relates to: FLINK-25872 Restoring from non-changelog checkpoint with changelog state-backend enabled in CLAIM mode discards state in use
• relates to: FLINK-28699 Native rocksdb full snapshot in non-incremental checkpointing
• links to: GitHub Pull Request #20484