[MAPREDUCE-1221] Kill tasks on a node if the free physical memory on that machine falls below a configured threshold

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.22.0
• Fix Version/s: 0.21.0
• Component/s: tasktracker
• Hadoop Flags: Reviewed
• Assignee: Scott Chen
• Reporter: Dhruba Borthakur
• Created: 19/Nov/09 00:48
• Updated: 24/Aug/10 21:19
• Resolved: 21/Apr/10 06:18

Description

The TaskTracker currently supports killing tasks if the virtual memory of a task exceeds a set of configured thresholds. I would like to extend this feature to enable killing tasks if the physical memory used by that task exceeds a certain threshold.

On a certain operating system (guess?), if user space processes start using lots of memory, the machine hangs and dies quickly. This means that we would like to prevent map-reduce jobs from triggering this condition. From my understanding, the killing-based-on-virtual-memory-limits (HADOOP-5883) were designed to address this problem. This works well when most map-reduce jobs are Java jobs and have well-defined -Xmx parameters that specify the max virtual memory for each task. On the other hand, if each task forks off mappers/reducers written in other languages (python/php, etc), the total virtual memory usage of the process-subtree varies greatly. In these cases, it is better to use kill-tasks-using-physical-memory-limits.