[FLINK-26853] HeapStateBackend ignores metadata updates in certain cases

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.14.4, 1.15.0, 1.16.0
• Fix Version/s: 1.16.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: Hangxiang Yu
• Reporter: Roman Khachatryan
• Created: 24/Mar/22 18:09
• Updated: 11/Oct/22 08:06
• Resolved: 26/Jul/22 15:44

Description

On recovery, HeapRestoreOperation reads state handles one by one;

• each handle contains metadata at the beginning;
• the metadata is always read, but not actually used if a state with the corresponding name was already registered

In a rare case of downscaling + multiple checkpoints with different metadata; this might lead to data being deserialized incorrectly (always using the initial metadata).

It also prevents incremental checkpoints with schema evolution.

On first access, however, the backend itself will update (merge) metadata; so that it doesn't affect new state updates.

Issue Links

• is duplicated by: FLINK-28553 The serializer in StateMap has not been updated when metaInfo of StateTable updated
• is related to: FLINK-23143 Support state migration
• relates to: FLINK-21648 FLIP-151: Incremental snapshots for heap-based state backend
• links to: GitHub Pull Request #20268