[BEAM-1250] Remove leaf when materializing PCollection to avoid re-evaluation.

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: runner-spark
• Assignee: Amit Sela
• Reporter: Amit Sela
• Created: 06/Jan/17 22:01
• Updated: 24/Jul/20 20:55
• Resolved: 07/Jan/17 09:09

Description

When materializing a PCollection (implemented as RDD), to create a PCollectionView for example, the runner should remove the materialized RDD from the "leaves" set.
The runner keeps track of leaves left un-handled in the DAG to force action on them - Write for one is implemented via a sequence of ParDos which are implemented by the runner via mapPartitions so we need to force an action.
Materializing an RDD is done via the action collect() so no reason to keep in "leaves" set.
Currently, it remains in the "leaves" set and so it is forced and evaluates the lineage and if not cached it will execute twice the lineage twice (unless caches are applied for some reason).

Issue Links

• links to: GitHub Pull Request #1747