[PIG-3856] UnionOptimizer in Tez should optimize the case of replicated join

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: tez
• Assignee: Unassigned
• Reporter: Rohini Palaniswamy
• Created: 31/Mar/14 18:58
• Updated: 30/Jan/15 22:29

Description

Replicate join input that was broadcast to union vertex now needs to be broadcast to all the union predecessors. So we need to

• Create edges from the Replicate join input to all the union predecessors
• Change replicate join input to write to multiple outputs.

This can be further optimized by using a shared edge which is yet to be implemented in Tez (TEZ-391)

Issue Links

• requires: TEZ-391 SharedEdge - Support for passing same output from a vertex as input to two different vertices
• requires: TEZ-1153 Allow object caching for more than one vertex