[FLINK-25842] [v2] FLIP-158: Generalized incremental checkpoints

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.18.0
• Component/s: Runtime / State Backends
• Assignee: Yuan Mei
• Reporter: Roman Khachatryan
• Created: 27/Jan/22 08:54
• Updated: 23/Jul/23 12:31
• Resolved: 23/Jul/23 10:17

Description

Umbrella ticket for the 2nd iteration of FLIP-158: Generalized incremental checkpoints 

1st iteration: FLINK-21352

======

Updated on July 23:

The second half of Changelog is done in the 1.17 release.

There are two remaining tasks related to 1) DFS writer benchmarking and 2) rescaling benchmarking, which I believe should not be a block for this. Moved to FLINK-32651

We've had thorough testing and performance evaluation. This feature is prod ready.

More details can be found in Flink 1.17 release note and this blog post

Issue Links

• causes: FLINK-26372 Allow to configure Changelog Storage per program
• causes: FLINK-23253 Expose changelog interfaces to enable custom implementations
• causes: FLINK-25255 Expose Changelog checkpoints via State Processor API
• causes: FLINK-24402 Add a metric for back-pressure from the ChangelogStateBackend
• is blocked by: FLINK-28976 Changelog 1st materialization delayed unneccesarily
• is caused by: FLINK-21352 FLIP-158: Generalized incremental checkpoints
• relates to: FLINK-22753 Activate log-based Checkpoints for StateFun
• relates to: FLINK-25512 Materialization Files are not cleaned up if no checkpoint is using it
• relates to: FLINK-24852 Cleanup of Orphaned Incremental State Artifacts
• relates to: FLINK-29082 Clean-up Leftovers for changelog pre-uploading files after failover
• relates to: FLINK-20103 Improve test coverage with chaos testing & side-by-side tests
• relates to: FLINK-25862 Refactor SharedStateRegistry to not limit StreamStateHandle to register/unregister
• relates to: FLINK-23035 Add explicit method to StateChangelogWriter to write metadata