[PIG-196] Pig should use '-reducer NONE' for map-only jobs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.0
• Assignee: Arun Murthy
• Reporter: Arun Murthy
• Created: 08/Apr/08 16:53
• Updated: 25/Mar/10 00:12
• Resolved: 30/Sep/08 23:07

Description

Currently, for map-only jobs, Pig writes map-outputs directly to HDFS and then sends zero data to reducers. The problem with this is two fold:

• Reduce slots are unnecessarily wasted on the cluster
• Reduces write empty files to HDFS putting pressure on the Namenode

Both these can we very easily avoided by just calling:

job.setNumReduces(0);

and letting Hadoop Map-Reduce take care of writing map-outputs directly to HDFS.

Issue Links

• is part of: PIG-157 Add types and rework execution pipeline