Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11146

Add option to disable copying between Flink runner

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: P2
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 2.26.0
    • Component/s: runner-flink
    • Labels:

      Description

      In order to implement Flink TypeSerializer the runner implements CoderTypeSerializer. The way the copy function is implemented is by first serializing and then deserializing each element. This means that such a deep copy needs to be done between each operator and this can become a bottleneck.

      The reason the copy functions need to be implemented is that Flink guarantees that elements will be deep copied between each operator. In Beam this is the users responsibility and so this is not strictly neccecarry.

      The aim of this improvement is to introduce an option on the Flink Runner, that eliminates this overhead, by simply returning the value.

      Here is the mailing list discussion

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rHermes Teodor Spæren
                Reporter:
                rHermes Teodor Spæren
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2.5h
                  2.5h