[FLINK-25842] [v2] FLIP-158: Generalized incremental checkpoints

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.18.0
• Component/s: Runtime / State Backends
• Assignee: Yuan Mei
• Reporter: Roman Khachatryan
• Created: 27/Jan/22 08:54
• Updated: 23/Jul/23 12:31
• Resolved: 23/Jul/23 10:17

Description

Umbrella ticket for the 2nd iteration of FLIP-158: Generalized incremental checkpoints 

1st iteration: FLINK-21352

======

Updated on July 23:

The second half of Changelog is done in the 1.17 release.

There are two remaining tasks related to 1) DFS writer benchmarking and 2) rescaling benchmarking, which I believe should not be a block for this. Moved to FLINK-32651

We've had thorough testing and performance evaluation. This feature is prod ready.

More details can be found in Flink 1.17 release note and this blog post

Issue Links

• causes: FLINK-26372 Allow to configure Changelog Storage per program
• causes: FLINK-23253 Expose changelog interfaces to enable custom implementations
• causes: FLINK-25255 Expose Changelog checkpoints via State Processor API
• causes: FLINK-24402 Add a metric for back-pressure from the ChangelogStateBackend
• is blocked by: FLINK-28976 Changelog 1st materialization delayed unneccesarily
• is caused by: FLINK-21352 FLIP-158: Generalized incremental checkpoints
• relates to: FLINK-22753 Activate log-based Checkpoints for StateFun
• relates to: FLINK-25512 Materialization Files are not cleaned up if no checkpoint is using it
• relates to: FLINK-24852 Cleanup of Orphaned Incremental State Artifacts
• relates to: FLINK-29082 Clean-up Leftovers for changelog pre-uploading files after failover
• relates to: FLINK-20103 Improve test coverage with chaos testing & side-by-side tests
• relates to: FLINK-25862 Refactor SharedStateRegistry to not limit StreamStateHandle to register/unregister
• relates to: FLINK-23035 Add explicit method to StateChangelogWriter to write metadata

Sub-Tasks

• Test Changelog StateBackend V2 Manually: Closed
• Add benchmarks for the ChangelogStateBackend: Resolved
• Support state migration: Closed
• Support local recovery: Closed
• Add/Expose/Differentiate metrics of checkpoint size between changelog size vs materialization size: Closed
• move “enablechangelog” constant out of flink-streaming-java module: Closed
• Expose changelog configuration in RestAPI and WebUI: Closed
• Restoring from non-changelog checkpoint with changelog state-backend enabled in CLAIM mode discards state in use: Closed
• Support recovery (from checkpoints) after disabling changelog backend: Closed
• Reduce multiple reads to the same Changelog file in the same taskmanager during restore: Closed
• Show the delegated StateBackend and whether changelog is enabled in the UI: Closed
• Pre-emptively uploaded changelog not discarded up if materialized before checkpoint: Closed
• [ZH] Add ChangelogBackend documentation: Closed
• Scatter dstl files into separate directories by job id: Closed
• State ownership: track and discard private state (materialized: rocks, heap): Closed
• State ownership: prevent shared state deletion on TM: Closed
• State ownership: compute shared state on recovery: Closed
• State ownership: track and discard private state (registry+changelog): Closed
• State ownership: prevent private state deletion on JM: Closed
• State ownership: Support more than one retained checkpoints: Closed
• State ownership: Consider disallowing in-memory state handles for materialized state: Closed