[PIG-4518] SparkOperator should correspond to complete Spark job

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Mohit Sabharwal
• Reporter: Mohit Sabharwal
• Created: 23/Apr/15 23:53
• Updated: 21/Jun/17 09:18
• Resolved: 29/Apr/15 00:45

Description

SparkPlan, which was added in PIG-4374, creates a new SparkOperator for every shuffle boundary (denoted by presence of POGlobalRearrange in the corresponding physical plan). This is unnecessary for Spark engine since it relies on Spark to do the shuffle (using groupBy(), reduceByKey() and CoGroupRDD) and does not need to explicitly identify "map" and "reduce" operations.

It is also cleaner if a single SparkOperator represents a single complete Spark job.

Issue Links

• links to: review board