[FLINK-25276] FLIP-203: Support native and incremental savepoints

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.3
• Fix Version/s: 1.15.0
• Component/s: Runtime / Checkpointing
• Assignee: Unassigned
• Reporter: Piotr Nowojski
• Created: 13/Dec/21 14:24
• Updated: 07/Apr/22 10:18
• Resolved: 07/Apr/22 10:18

Description

Motivation. Currently with non incremental canonical format savepoints, with very large state, both taking and recovery from savepoints can take very long time. Providing options to take native format and incremental savepoint would alleviate this problem.

In the past the main challenge lied in the ownership semantic and files clean up of such incremental savepoints. However with FLINK-25154 implemented some of those concerns can be solved. Incremental savepoint could leverage "force full snapshot" mode provided by FLINK-25192, to duplicate/copy all of the savepoint files out of the Flink's ownership scope.

Issue Links

• relates to: FLINK-26326 FLIP-203: Support native and incremental savepoints 1.1
• relates to: FLINK-6755 Allow triggering Checkpoints through command line client
• relates to: FLINK-26147 Incremental savepoints directory in CLAIM mode is not being deleted
• relates to: FLINK-24149 Make checkpoint self-contained and relocatable
• supercedes: FLINK-12619 Support TERMINATE/SUSPEND Job with Checkpoint

Sub-Tasks

• Support native savepoints (w/o modifying the statebackend specific snapshot strategies): Resolved
• Support RocksDB incremental native savepoints: Closed
• Add tests for native savepoint format schema evolution: Closed
• Update chinese documentation regarding FLIP-203: Resolved
• Document checkpoint/savepoint guarantees: Closed
• Add test coverage for native format job upgrades: Closed
• Add test coverage for native format flink upgrades (minor versions): Closed
• Reintroduce old JobClient#triggerSavepoint,stopWithSavepoint methods: Closed
• Add documentation of savepoint type on Command-Line part: Closed