[PIG-2784] Framework for dynamic query optimization

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: GSOC2014
• Assignee: Unassigned
• Reporter: Jie Li
• Created: 02/Jul/12 20:45
• Updated: 12/May/14 18:17

Description

We need a framework to implement dynamic query optimization, i.e. changing the query plan at runtime. Currently we support estimating the number of reducers dynamically, which works well as the first step but was not perfectly implemented. In near future, we'll support more dynamic optimization, like removing sample job for order-by, removing limit job, dynamically detecting skew and using skew-join, etc.

Currently estimating #reducer is implemented in JobControlCompiler after MRCompiler compiles all the MapReduceOperators and generate the complete MRPlan. One place (discussed with Thejas) to implement the framework is at the MRCompiler, where the MRPlan'll be generated at batches and adjusted dynamically.

Any comment?

This is a candidate project for Google summer of code 2014. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2014

Issue Links

• is related to: PIG-2772 Convert the skew join to the normal join for small dataset
• is related to: PIG-483 PERFORMANCE: different strategies for large and small order bys
• is related to: PIG-2675 Optimization: Remove unnecessary Limit jobs from plan
• relates to: PIG-2779 Refactoring the code for setting number of reducers