[FLINK-25458] Support local recovery

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Runtime / Checkpointing, Runtime / State Backends
• Assignee: Yanfei Lei
• Reporter: Yun Tang
• Created: 27/Dec/21 09:51
• Updated: 09/Aug/22 10:33
• Resolved: 09/Aug/22 10:33

Description

Currently, changelog state-backend doesn't support local recovery. Thus, recovery times might be sub-optimal.

 

Materialized state issues:

Current periodic materialization would call state backend snapshot method with a materialization id. However, current local state managment would rely on checkpoint id as storing, confirming and discarding. The gap between them would break how local recovery works.

 

Non-materialized state issues:

• non-materialized state (i.e. changelog) is shared across checkpoints, and therefore needs some tracking (in TM or hard-linking in FS)
• the writer does not enforce boundary between checkpoints (when writing to DFS); if local stream simply duplicates DFS stream then it would break on cleanup
• files can be shared across tasks, which will also break on cleanup

Issue Links

• is a parent of: FLINK-27692 Support local recovery for materialized part(write, restore, discard)
• is a parent of: FLINK-27693 Support local recovery for non-materialized part(write, restore, discard)
• relates to: FLINK-28286 move “enablechangelog” constant out of flink-streaming-java module