[FLINK-8547] Implement CheckpointBarrierHandler not to spill data for exactly-once based on credit-based flow control

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: Runtime / Network
• Assignee: Zhijiang
• Reporter: Zhijiang
• Created: 02/Feb/18 07:42
• Updated: 22/Feb/18 13:31
• Resolved: 22/Feb/18 13:30

Description

Currently in exactly-once mode, the BarrierBuffer would block inputs with barriers until all inputs have received the barrier for a given checkpoint. To avoid back-pressuring the input streams which may cause distributed deadlocks, the BarrierBuffer has to spill the data in disk files to recycle the buffers for blocked channels.

 

Based on credit-based flow control, every channel has exclusive buffers, so it is no need to spill data for avoiding deadlock. Then we implement a new CheckpointBarrierHandler for only buffering the data for blocked channels for better performance.

 

And this new CheckpointBarrierHandler can also be configured to use or not in order to rollback the original mode for unexpected risks.

Issue Links

• links to: GitHub Pull Request #5400