[PIG-3658] Use Tez ObjectRegistry to cache FRJoin map and WeightedRangePartitioner map

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: tez-branch
• Component/s: tez
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 08/Jan/14 17:24
• Updated: 21/Nov/14 05:58
• Resolved: 27/Jan/14 20:50

Description

Tez provides way to cache objects with Vertex scope. We can use it to
1) caching the replication join map constructed instead of reading from multiple broadcast inputs from one or more vertex and reconstructing it
2) Cache the map in WeightedRangePartitioner. Thought the quantilefile is only one brodcast input from one vertex still it would be faster to use the local cache.

Issue Links

• requires: TEZ-711 Fix memory leak when not reading from inputs due to caching