[FLINK-19297] Make ResultPartitionWriter record-oriented

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.12.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yingjie Cao
• Reporter: Yingjie Cao
• Created: 21/Sep/20 03:31
• Updated: 30/Sep/20 09:34
• Resolved: 25/Sep/20 02:25

Description

Currently, ResultPartitionWriter is buffer-oriented, that is, RecordWriter will add buffer of different channels to ResultPartitionWriter and the buffer boundary serves as a nature boundary of data belonging to different channels. However, this abstraction is not flexible enough to handle some cases where records are appended a joint-structure shared by all channels and sorting is used to cluster data belonging to different channels.

In this ticket, we propose to make ResultPartitionWriter record oriented which offers more flexibility to the implementation of ResultPartitionWriter. And based on the new record-oriented Interface, we will introduce the sort-merge based blocking shuffle to Flink in the future.

Issue Links

• causes: FLINK-19441 Performance regression on 24.09.2020
• links to: GitHub Pull Request #13447

Sub-Tasks

• Flushing of BoundedBlockingResultPartition should finish current BufferBuilder: Closed
• Introduce BufferWritingResultPartition which wraps the logic of writing buffers to ResultSubpartition: Closed
• Remove RecordWriter#clearBuffers: Closed
• Small optimization of network layer record serialization: Closed