[FLINK-21352] FLIP-158: Generalized incremental checkpoints

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Runtime / Checkpointing, Runtime / State Backends
• Assignee: Roman Khachatryan
• Reporter: Roman Khachatryan
• Created: 10/Feb/21 16:45
• Updated: 08/May/22 15:53
• Resolved: 08/Mar/22 07:22

Description

Umbrella ticket for FLIP-158: Generalized incremental checkpoints (v1).

[v2] FLIP-158: Generalized incremental checkpoints can be found FLINK-25842

Issue Links

• blocks: FLINK-22753 Activate log-based Checkpoints for StateFun
• causes: FLINK-25842 [v2] FLIP-158: Generalized incremental checkpoints
• is blocked by: FLINK-26079 [Changelog] Disallow recovery from non-changelog checkpoints
• is blocked by: FLINK-26231 [Changelog] Incorrect MaterializationID passed to ChangelogStateBackendHandleImpl
• is blocked by: FLINK-25261 Changelog not truncated on materialization
• is blocked by: FLINK-25524 If enabled changelog, RocksDB incremental checkpoint would always be full
• is blocked by: FLINK-26396 [Changelog] Upload is not failed even if all attempts timeout
• is blocked by: FLINK-26255 SplitAggregateITCase.testAggWithJoin failed on azure
• is related to: FLINK-25260 Recovery fails when using changelog+s3+presto
• is related to: FLINK-23441 Remove CheckpointOptions Argument away from Snapshotable#snapshot
• relates to: FLINK-23534 Banned dependencies in flink-statebackend-changelog
• relates to: FLINK-25144 Test Changelog State Backend manually
• mentioned in: Page Loading...

Sub-Tasks

• Same materialized state handle should not register multi times: Resolved
• Randomize periodic materialisation interval in tests: Resolved
• Add ITCase for Generalized incremental checkpoints: Resolved
• Add ChangelogBackend documentation: Resolved
• Include changelog jars into distribution: Resolved
• Prevent JM from discarding state on checkpoint abortion: Resolved
• Write metadata after materialization: Closed
• State ownership: notify TMs about checkpoint subsumption: Resolved
• Provide backpressure (currently job fails if a limit is hit): Resolved
• Add metrics for the Changelog uploader: Resolved
• Add periodic materialization: Closed
• Revert FLINK-23738 (i.e. unhide config, API, docs): Closed
• Hide any configuration, API or docs: Closed
• Enable changelog backend in tests: Closed
• Add percentiles to checkpoint stats: Closed
• Cleanup unnecessary dependencies in dstl pom.xml: Closed
• Optimize writing state changes: Closed
• Add FS-based StateChangelog implementation: Closed
• Add StateChangelog interface and its in-memory implementation: Closed
• Provide StateBackends access to MailboxExecutor: Closed
• Add ChangelogStateBackend (proxy-everything): Closed
• Send changes to the state changelog (still proxy everything): Closed
• Implement incremental checkpointing and recovery using state changelog: Closed
• Test Changelog State backend (Wrapper): Resolved
• Add Changelog state for timers (PQ): Closed
• Create and wire changelog writer with backend: Closed
• Log metadata: Closed
• ChangelogStateBackend tests use nested backend on recovery: Closed
• Adjust configuration of ChangelogStateBackend and hide it from users: Resolved
• Python API for enabling ChangelogStateBackend: Closed
• testKeyGroupedInternalPriorityQueue does not dispose ChangelogDelegateEmbeddedRocksDB properly, and fails the test: Closed
• Release TaskManagerJobMetricGroup with the last slot rather than task: Resolved
• Introduce incremental/full checkpoint size stats: Resolved