[MAPREDUCE-3205] MR2 memory limits should be pmem, not vmem

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.0
• Component/s: mrv2, nodemanager
• Target Version/s: 0.23.0
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Resource limits are now expressed and enforced in terms of physical memory, rather than virtual memory. The virtual memory limit is set as a configurable multiple of the physical limit. The NodeManager's memory usage is now configured in units of MB rather than GB.

• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 19/Oct/11 01:14
• Updated: 15/Nov/11 00:49
• Resolved: 26/Oct/11 23:58

Description

Currently, the memory resources requested for a container limit the amount of virtual memory used by the container. On my test clusters, at least, Java processes take up nearly twice as much vmem as pmem - a Java process running with -Xmx500m uses 935m of vmem and only about 560m of pmem.

This will force admins to either under-utilize available physical memory, or oversubscribe it by configuring the available resources on a TT to be larger than the true amount of physical RAM.

Instead, I would propose that the resource limit apply to pmem, and allow the admin to configure a "vmem overcommit ratio" which sets the vmem limit as a function of pmem limit.