[FLINK-23461] State ownership: Consider disallowing in-memory state handles for materialized state

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 1.16.0
• Component/s: Runtime / State Backends
• Assignee: Unassigned
• Reporter: Roman Khachatryan
• Created: 21/Jul/21 16:07
• Updated: 15/Aug/22 14:25
• Resolved: 15/Aug/22 14:25

Description

For non-mateialized part, FLINK-21353 doesn't use nor FsCheckpointStreamFactory neither PlaceholderStreamStateHandle so it's not an issue. Adding it in the future doesn't make sense as for such small changes incremental checkpoint might work better.

 

For materialized part, ByteStreamStateHandle can be currently used. This can bring back to life issues like FLINK-21351 - if checkpoint subsumption on TM will be decoupled from the state backends state. Removing those assumptions is one of the goals of changing the ownership.
An easy way to solve it is to just enforce zero threshold for writing to DFS instead of memory.

 

PlaceholderStreamStateHandle can be used for the materialized state (regardless of ByteStreamStateHandle; unless SnapshotStrategy is changed). However, it shouldn't cause any issues:

• if the file is shared (i.e. after recovery) then by definition it should be managed by JM
• otherwise, JM should still replace placeholders (FLINK-23137); and it should have received the original state objects before; no re-upload should happen (FLINK-23344) - so JM and TM will always refer to the same file