[FLINK-17972] Consider restructuring channel state

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.11.0
• Component/s: Runtime / Checkpointing
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
• Assignee: Unassigned
• Reporter: Roman Khachatryan
• Created: 27/May/20 12:59
• Updated: 11/Dec/23 07:35

Description

(depends on rescaling for unaligned checkpoints (FLINK-17979))

 

Current structure is the following (this PR doesn't change it):

Each subtask reports to JM TaskStateSnapshot
  each with zero ore more OperatorSubtaskState,
    each with zero or more InputChannelStateHandle and ResultSubpartitionStateHandle
      each referencing an underlying StreamStateHandle

The underlying StreamStateHandle duplicates filename (ByteStreamStateHandle has it too at least because of equals/hashcode I guess).

An alternative would be something like

Each subtask reports to JM TaskStateSnapshot
  each with zero ore more OperatorSubtaskState
    each with zero or one StreamStateHandle (for channel state)
    each with zero or more InputChannelStateHandle and ResultSubpartitionStateHandle

(probably, with StreamStateHandle and InputChannelStateHandle and ResultSubpartitionStateHandle encapsulated)

 

It would be more effective (less data duplication) but probably also more error-prone (implicit structure), less flexible (re-scaling).

(as discussed during introduction of StreamStateHandle.asBytesIfInMemory here)
 

Issue Links

• is blocked by: FLINK-17979 Support rescaling for Unaligned Checkpoints