Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10745 Serialization and copy improvements for record writer
  3. FLINK-10995

Copy intermediate serialization results only once for broadcast mode

    XMLWordPrintableJSON

    Details

      Description

      The emitted records from operator would be firstly serialized into intermediate bytes array in RecordSerializer, then copy the intermediate results into target buffers for different sub partitions.  For broadcast mode, the same intermediate results would be copied as many times as the number of sub partitions, and this would affect the performance seriously in large scale jobs.

      We can copy to only one target buffer which would be shared by all the sub partitions to reduce the overheads. For emitting latency marker in broadcast mode, we should flush the previous shared target buffers first, and then request a new buffer for the target sub partition to send latency marker.

        Attachments

          Activity

            People

            • Assignee:
              zjwang zhijiang
              Reporter:
              zjwang zhijiang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m