[FLINK-14396] Implement rudimentary non-blocking network output

• Type: Sub-task
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.10.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Zhijiang
• Reporter: Zhijiang
• Created: 15/Oct/19 11:29
• Updated: 14/Oct/20 07:25
• Resolved: 28/Oct/19 06:16

Description

Considering the mailbox model and unaligned checkpoints requirements in future, task network output should be non-blocking. In other words, as long as output is available, it should never block for a subsequent/future single record write.

In the first version, we only implement the non-blocking output for the most regular case, and do not solve the following cases which still keep the previous behavior.

• Big record which might span multiple buffers
• Flatmap-like operators which might emit multiple records in every process
• Broadcast watermark which might request multiple buffers at a time

The solution is providing the RecordWriter#isAvailable method and respective LocalBufferPool#isAvailable for judging the output beforehand. As long as there is at-least one available buffer in LocalBufferPool, the RecordWriter is available for network output in most cases.  This doesn’t include runtime handling of this non-blocking and availability behavior in StreamInputProcessor

Note: It requires the minimum number of buffers in output LocalBufferPool adjusting to (numberOfSubpartitions + 1) and also adjusting the monitor of backpressure future.

Issue Links

• links to: GitHub Pull Request #9905