Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7989

SparkRunner CacheVisitor counts PCollections from SideInputs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.14.0
    • Fix Version/s: 2.16.0
    • Component/s: runner-spark
    • Labels:
      None

      Description

      The SparkRunner's CacheVisitor looks at all inputs for a TransformHierarchy.Node. Those inputs include the PCollections from the PCollectionViews that are supplied as sideInputs.

      The SparkRunner should not count these instances of sideInputs as the PCollections are not actually accessed. They are only accessed when the CreatePCollectionView Transform is processed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                winkelman.kyle Kyle Winkelman
                Reporter:
                winkelman.kyle Kyle Winkelman
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m