[PIG-4709] Support combine for spark mode

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Labels: spork
• Assignee: Pallavi Rao
• Reporter: Pallavi Rao
• Created: 21/Oct/15 08:05
• Updated: 21/Jun/17 09:18
• Resolved: 29/Jan/16 03:45

Description

Currently, the GROUPBY operator of PIG is mapped by Spark's CoGroup. When the grouped data is consumed by subsequent operations to perform algebraic operations, this is sub-optimal as there is lot of shuffle traffic.

The Spark Plan must be optimized to use reduceBy, where possible, so that a combiner is used.

Issue Links

• is required by: PIG-4766 Ensure GroupBy is optimized for all algebraic Operations
• links to: Review Board