[FLINK-10745] Serialization and copy improvements for record writer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Component/s: Runtime / Network, Runtime / Task
• Assignee: Zhijiang
• Reporter: Zhijiang
• Created: 01/Nov/18 08:56
• Updated: 27/Sep/19 15:15
• Resolved: 27/Sep/19 15:14

Description

This is the umbrella issue for tracking the improvements of serialization and copy during emitting records.

 

There are mainly involved in three components:

1. ChannelSelector : Refactor the interface for adding setup method and changing selectChannel for single channel.

2. RecordSerializer : Redesign the processes for serialization and copy only once for multiple selected channels.

2. RecordWriter : Redesign the process for handling broadcast mode in shortcut way.

 

The launched discussion is under http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improve-broadcast-serialization-td23295.html

Sub-Tasks

• Make the condition of broadcast partitioner simple: Closed
• Refactor the ChannelSelector interface for single selected channel: Closed
• Improve output serialization only once in RecordWriter: Closed
• Network throughput performance regression after broadcast changes: Closed
• Refactor all the StreamPartitioner implementations into runtime module: Closed
• Abstract the StreamPartitionerTest for common codes: Closed
• Simplify the RebalancePartitioner implementation: Closed
• Remove duplicated Round-robin ChannelSelector implementation in RecordWriterTest: Closed
• Deduplicate common codes in OutputEmitterTest: Closed
• Copy intermediate serialization results only once for broadcast mode: Closed
• Merge StreamRecordWriter into RecordWriter: Closed
• Introduce RecordWriterBuilder for creating RecordWriter instance: Closed
• Remove Optional fields from RecordWriter relevant classes: Resolved