[FLINK-16428] Fine-grained network buffer management for backpressure

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: Runtime / Network
• Assignee: Unassigned
• Reporter: Zhijiang
• Created: 04/Mar/20 15:40
• Updated: 01/Sep/21 10:52
• Resolved: 13/Jul/21 07:59

Description

It is an umbrella ticket for tracing the progress of this improvement.

This is the second ingredient to solve the “checkpoints under backpressure” problem (together with unaligned checkpoints). It consists of two steps:

• See if we can use less network memory in general for streaming jobs (with potentially different distribution of floating buffers in the input side)
• Under backpressure, reduce network memory to have less in-flight data (needs specification of algorithm and experiments)

Sub-Tasks

• Reduce the default number of exclusive buffers from 2 to 1 on receiver side: Closed
• Avoid caching buffers for blocked input channels before barrier alignment: Closed
• Announce sender's backlog to solve the deadlock issue without exclusive buffers: Closed
• Limit the maximum backlogs in subpartitions for data skew case: Closed
• Add checkpoint with remote channel benchmark: Closed
• Remove checkpoint alignment buffered metric from webui: Closed
• Make network buffers per incoming/outgoing channel can be configured separately: Closed