[FLINK-23252] Support recovery (from checkpoints) after disabling changelog backend

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Release Note: Added support for disabling changelog when recovering from checkpoints
• Assignee: Hangxiang Yu
• Reporter: Roman Khachatryan
• Created: 05/Jul/21 11:13
• Updated: 08/Aug/22 21:52
• Resolved: 25/Apr/22 06:29

Description

Recovery from savepoint already works:

• changelog off -> on: FLINK-23278
• changelog on -> off: directly calling underlyingBackend.savepoint() (instead of writer.persist()) 

 

For checkpoints, changelog on -> off doesn't work. Regular state backend will receive materialized and non-aterialized state on recovery. To support it, state changes have to be applied on recovery even if changelog backend is disabled.

Another obstacle could be that wrapped backend state handles created during materialization can refer to a fake checkpoint ID. This fake checkpoint ID would then be passed to e.g. RocksIncrementalSnapshotStrategy on restore.

Issue Links

• links to: GitHub Pull Request #19142
• links to: GitHub Pull Request #20486