[FLINK-17122] Support InputSelectable and BoundedMultiInput operators with checkpointing

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.9.2, 1.10.0
• Component/s: Runtime / Checkpointing, Runtime / Network
• Labels: auto-deprioritized-major auto-deprioritized-minor
• Assignee: Unassigned
• Reporter: Piotr Nowojski
• Created: 13/Apr/20 12:14
• Updated: 30/Dec/21 15:13

Description

Currently when user defined some InputSelectable or BoundedMultiInput operators, checkpointing is not supported. Main problem is the that combination of InputSelectable and barrier alignment can lead to deadlocks (checkpoint barrier stuck on not selected channel).

Problem could be somehow mitigated via unaligned checkpoints (FLINK-14551), but not fully. Even with unaligned checkpoints, checkpoint barriers can be stuck in the job graph if there is a flatMap operator (or non flatMap operator but if records are spanning multiple buffers), blocked in the middle of processing by some down stream input selection. In such case we are not able to perform unaligned checkpoint.

Potential solution could be using persistent communication channels or detecting before mentioned situations and avoid the dead lock by spilling excess data.

As most of the problems are with input selections that are randomly flipping, there might be some partial solution for more trivial cases, like reading one side of an input fully before the other.

Issue Links

• is duplicated by: FLINK-25424 Checkpointing is currently not supported for operators that implement InputSelectable
• is related to: FLINK-14551 Unaligned checkpoints
• is related to: FLINK-19113 Add support for checkpointing with selectable inputs
• relates to: FLINK-21392 Support to block in connected streams