[HADOOP-4523] Enhance how memory-intensive user tasks are handled

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.19.0
• Fix Version/s: 0.20.0
• Hadoop Flags: Reviewed
• Assignee: Vinod Kumar Vavilapalli
• Reporter: Vivek Ratan
• Created: 27/Oct/08 06:47
• Updated: 02/May/13 02:29
• Resolved: 19/Nov/08 06:23

Description

HADOOP-3581 monitors each Hadoop task to see if its memory usage (which includes usage of any tasks spawned by it and so on) is within a per-task limit. If the task's memory usage goes over its limit, the task is killed. This, by itself, is not enough to prevent badly behaving jobs from bringing down nodes. What is also needed is the ability to make sure that the sum total of VM usage of all Hadoop tasks does not exceed a certain limit.

Issue Links

• depends upon: HADOOP-4035 Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements and task trackers free memory