[FLINK-11035] Notify data available to network stack immediately after finishing BufferBuilder

• Type: Sub-task
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Component/s: Runtime / Network
• Assignee: Zhijiang
• Reporter: Zhijiang
• Created: 30/Nov/18 07:58
• Updated: 21/Jul/20 06:55
• Resolved: 21/Jul/20 06:55

Description

The data availability notification for network relies on whether there are finished BufferBuilder or flush triggered. If flush is not triggered and the first BufferBuilder enqueues into the subpartition, although this BufferBuilder is finished on RecordWriter side, it has to rely on enqueuing the second BufferBuilder to trigger notification available.  It may bring some delays for transporting the finished BufferBuilder in network, especially there has a blocking operation for requesting the second BufferBuilder from pool.

Supposing there is only one available buffer in LocalBufferPool in extreme scenarios, if the first BufferBuilder is not transported and recycled, the requesting for second BufferBuilder will be blocked all the time.

I propose to add a notifyBufferFinished method in ResultPartitionWriter interface, then RecordWriter can notify via it after BufferBuilder finished_._