[PIG-2779] Refactoring the code for setting number of reducers

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11
• Assignee: Jie Li
• Reporter: Jie Li
• Created: 29/Jun/12 02:03
• Updated: 22/Feb/13 04:53
• Resolved: 31/Jul/12 17:36

Description

As PIG-2652 observed, currently the code for setting number of reducers is a little messy. MapReduceOper.requestedParallelism seems being misused in some plases, and now we support runtime estimation of #reducer which further complicates the problem.

For example, if we specify parallel 1 for the order-by, the estimated #reducer will be used. If we specify parallel 2 while it estimates 4, order-by will fail due to "Illegal partition for Null". If we specify parallel 4 while it estimates 2, then some reducers will have nothing to do.

Issue Links

• blocks: PIG-2772 Convert the skew join to the normal join for small dataset
• blocks: PIG-483 PERFORMANCE: different strategies for large and small order bys
• is related to: PIG-2652 Skew join and order by don't trigger reducer estimation
• is related to: PIG-2784 Framework for dynamic query optimization