[MAPREDUCE-1521] Protection against incorrectly configured reduces

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.22.1
• Component/s: jobtracker
• Assignee: Mahadev Konar
• Reporter: Arun Murthy
• Created: 22/Feb/10 17:43
• Updated: 29/Jul/14 23:35
• Resolved: 29/Jul/14 23:35

Description

We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.

This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.

Proposal: Come up with heuristics/configs to fail such jobs early.

Thoughts?

Issue Links

• is related to: PIG-1249 Safe-guards against misconfigured Pig scripts without PARALLEL keyword