[PIG-3618] Replace broadcast edges with scatter/gather edges in union

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: tez-branch
• Fix Version/s: tez-branch
• Component/s: tez
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 10/Dec/13 20:52
• Updated: 21/Nov/14 05:59
• Resolved: 11/Dec/13 22:58

Description

Previously, I implemented union using OnFileUnorderedKVOutput + broadcast edge. But this is a misuse of broadcast edge since union will create duplicate records when parallel is set to more than 1. We should replace them with ShuffledMergedInput + scatter/gather edge having the entire record as key.

Ideally, we should implement union using OnFileUnorderedKVOutput + scatter/gather edge with a round robin partitioner. For now, this is not supported by Tez (TEZ-661).