[FLINK-27251] Timeout aligned to unaligned checkpoint barrier in the output buffers of an upstream subtask

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0, 1.15.0
• Fix Version/s: 1.16.0
• Component/s: Runtime / Checkpointing
• Labels: pull-request-available
• Assignee: Rui Fan
• Reporter: Rui Fan
• Created: 14/Apr/22 14:10
• Updated: 20/Apr/23 11:49
• Resolved: 25/May/22 06:33

Description

After FLINK-23041, the downstream task can be switched UC when currentTime - triggerTime > timeout. But the downstream task still needs wait for all barriers of upstream. 

If the back pressure is serve, the downstream task cannot receive all barrier within CP timeout, causes CP to fail.

 

Can we support upstream Task switching from Aligned to UC? It means that when the barrier cannot be sent from the output buffer to the downstream task within the execution.checkpointing.aligned-checkpoint-timeout, the upstream task switches to UC and takes a snapshot of the data before the barrier in the output buffer.

 

Hi akalashnikov , please help take a look in your free time, thanks a lot.

Issue Links

• causes: FLINK-28077 Tasks get stuck during cancellation in ChannelStateWriteRequestExecutorImpl
• causes: FLINK-31588 The unaligned checkpoint type is wrong at subtask level
• causes: FLINK-31864 Generate unaligned checkpoint type based on persisted data
• relates to: FLINK-31588 The unaligned checkpoint type is wrong at subtask level
• links to: GitHub Pull Request #19723