Details
-
Sub-task
-
Status: Closed
-
Minor
-
Resolution: Fixed
-
1.8.0, 1.9.0
Description
The emitted records from operator would be firstly serialized into intermediate bytes array in RecordSerializer, then copy the intermediate results into target buffers for different sub partitions. For broadcast mode, the same intermediate results would be copied as many times as the number of sub partitions, and this would affect the performance seriously in large scale jobs.
We can copy to only one target buffer which would be shared by all the sub partitions to reduce the overheads. For emitting latency marker in broadcast mode, we should flush the previous shared target buffers first, and then request a new buffer for the target sub partition to send latency marker.
