[PIG-792] PERFORMANCE: Support skewed join in pig

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Assignee: Sriranjan Manjunath
• Reporter: Sriranjan Manjunath
• Created: 30/Apr/09 07:22
• Updated: 25/Mar/10 00:12
• Resolved: 30/Jul/09 00:31

Description

Fragmented replicated join has a few limitations:

• One of the tables needs to be loaded into memory
• Join is limited to two tables

Skewed join partitions the table and joins the records in the reduce phase. It computes a histogram of the key space to account for skewing in the input records. Further, it adjusts the number of reducers depending on the key distribution.

We need to implement the skewed join in pig.

Issue Links

• is cloned by: PIG-929 Default value of memusage for skewed join is not correct
• is cloned by: PIG-954 Skewed join fails when pig.skewedjoin.reduce.memusage is not configured
• is cloned by: PIG-955 Skewed join generates incorrect results
• is cloned by: PIG-1135 skewed join partitioner returns negative partition index