[PIG-4079] Parallel clause is not honored in local mode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: impl
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 28/Jul/14 20:36
• Updated: 21/Nov/14 05:58
• Resolved: 29/Jul/14 05:24

Description

I want to debug my custom partitioner in local mode. But it's not possible with Pig because:

1. MR does not invoke partitioner if # of reducers is equal to 1.
   https://issues.apache.org/jira/browse/MAPREDUCE-1287
2. Pig always forces a single reducers in local mode.
   LogicalPlanBuilder.java

        void setParallel(LogicalRelationalOperator op, Integer parallel) {
            if( parallel != null ) {
               op.setRequestedParallelism( pigContext.getExecType() == ExecType.LOCAL ? 1 : parallel );
            }
        }
   

After I change Pig to honor the parallel clause, my custom partitioner gets invoked in local mode.

Issue Links

• breaks: PIG-4089 TestMultiQuery.testMultiQueryJiraPig1169 fails in trunk after PIG-4079 in Hadoop 1