[FLINK-20976] Unify binary format for Keyed State savepoints - ASF JIRA

XML

Word

Printable

JSON

Release Note:

Hide
Flink’s savepoint binary format is unified across all state backends. That means you can take a savepoint with one state backend and then restore it using another.

If you want to switch the state backend you should first upgrade your Flink version to 1.13 then take a savepoint with the new version, and only after that, you can restore it with a different state backend.

Show
Flink’s savepoint binary format is unified across all state backends. That means you can take a savepoint with one state backend and then restore it using another. If you want to switch the state backend you should first upgrade your Flink version to 1.13 then take a savepoint with the new version, and only after that, you can restore it with a different state backend.

The main goal of this proposal is the following:

Unify across all state backends a savepoint format for keyed state that is more future-proof and applicable for potential new state backends. Checkpoint formats, by definition, are still allowed to be backend specific.
Make it possible to switch a state backend via a savepoint
Rework abstractions related to snapshots and restoring, to reduce the overhead and code duplication when attempting to implement a new state backend.

fixes

FLINK-4493 Unify the snapshot output format for keyed-state backends

FLINK-18165 When savingpoint is restored, select the checkpoint directory and stateBackend

supercedes

FLINK-11254 Unify serialization format of savepoint for switching state backends

1.	Implement HeapSavepointRestoreOperation	Closed	Dawid Wysakowicz
2.	Refactor the SnapshotStrategy hierarchy	Closed	Dawid Wysakowicz
3.	Extract common full-snapshot writer from RocksDB full-snapshot strategy	Closed	Aljoscha Krettek
4.	Make StateTable snapshots iterable	Closed	Dawid Wysakowicz
5.	Make HeapPriorityQueueStateSnapshot iterable	Closed	Dawid Wysakowicz
6.	Write savepoints in unified format from HeapStateBackend	Closed	Dawid Wysakowicz
7.	Update documentation with possible migration strategies	Closed	Dawid Wysakowicz
8.	Support switching from/to rocks db with heap timers	Closed	Dawid Wysakowicz
9.	Add sanity check when switching from Rocks to Heap timers	Closed	Dawid Wysakowicz
10.	Enforce common savepoint format at the operator level	Closed	Aljoscha Krettek