[PIG-3835] Improve performance of union

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: tez-branch
• Fix Version/s: tez-branch
• Component/s: tez
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Cheolsoo Park
• Created: 25/Mar/14 22:26
• Updated: 21/Nov/14 05:59
• Resolved: 02/Apr/14 14:30

Description

PIG-3743 implements union using VertexGroup. But there are a couple of optimizations that we can apply to it.

• Union followed by store
  Union is a blocking operator meaning that a new vertex is added for its succeeding operators. But if there is only one store in the succeeding vertex, MROutput could be directly attached to VertexGroup instead of adding a new vertex for it. Then, each union source vertex will write directly to the destination, and therefore, it will be faster.

• Replace POLocalRearrangeTez with POValueOutputTez
  Union uses POLocalRearrange by setting the whole record as key. But since union only needs to partition records evenly across tasks, it might make more sense to use POValueOutputTez with RR partitioner instead.

Issue Links

• is related to: PIG-3743 Use VertexGroup and Alias vertex for union
• relates to: PIG-3855 Turn on UnionOptimizer by default and add new e2e tests for union
• requires: TEZ-1003 Need a input that merges multiple ShuffleMergedInput from VertexGroup