[PIG-483] PERFORMANCE: different strategies for large and small order bys

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.2.0
• Labels: gsoc2011 performance
• Assignee: Jie Li
• Reporter: Olga Natkovich
• Created: 09/Oct/08 19:05
• Updated: 28/Sep/12 04:29

Description

Currently pig always does a multi-pass order by where it first determines a distribution for the keys and then orders in a second pass. This avoids the necessity of having a single reducer. However, in cases where the data is small enough to fit into a single reducer, this is inefficient. For small data sets it would be good to realize the small size of the set and do the order by in a single pass with a single reducer.

This is a candidate project for Google summer of code 2011. More information about the program can be found at http://wiki.apache.org/pig/GSoc2011

Issue Links

• is blocked by: PIG-2779 Refactoring the code for setting number of reducers
• is related to: PIG-2675 Optimization: Remove unnecessary Limit jobs from plan
• relates to: PIG-2772 Convert the skew join to the normal join for small dataset
• relates to: PIG-2784 Framework for dynamic query optimization