[PIG-3631] Improve performance of replicate-join

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: tez
• Assignee: Unassigned
• Reporter: Rohini Palaniswamy
• Created: 17/Dec/13 15:33
• Updated: 09/Oct/14 20:30

Description

Replicated join is implemented in Tez as follows:

• POFRJoinTez extends POFRJoin. The difference between two is that replication hash table is constructed out of broadcasting edges in Tez instead of files on distributed cache in MR.
• TezCompiler adds a vertex per replicated table and connect it to POFRJoin vertex via broadcasting edge.

Verify no performance regression with MR:

• The above approach is good when the replicate join is not the first vertex of the DAG (i.e in case of a MR, replicate join is part of a reduce). If it is the first vertex of the DAG, we need to compare and see that with this approach the performance does not regress with the MR's map only replicate join using distributed cache.

Evaluate:

• Instead of broadcasting key values and constructing hashmap, evaluate broadcasting (or distributing via cache based on performance) serialized hashmap and loading it as is similar to hive.