[PIG-2728] Grunt shell automagically sets number of reducers and doesn't allow you to override it

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.8.1
• Component/s: grunt

Environment

CentOS
12 boxes
30 map/reduce tasks max per box @ 2GB memory each

• Assignee: Unassigned
• Reporter: Lucas Lemanowicz
• Created: 29/May/12 21:23
• Updated: 01/Nov/12 18:50
• Resolved: 01/Nov/12 18:50

Description

When running Pig scripts through the grunt shell I kept getting Java Heap Space error messages. The cause was that my Hadoop cluster was running 467 map tasks which were being passed down to only 199 reducers. The notice I was getting through the Grunt shell was:

"Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 199"

I tried the following commands, none of which worked:

SET DEFAULT_PARALLELISM 2;
SET mapred.reduce.tasks 450;
SET mapred.running.reduce.limit 30;

When I ran the script directly from a file (pig filename.pig) it ran without problems.