[YARN-4768] getAvailablePhysicalMemorySize can be inaccurate on linux

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.2, 3.0.0-alpha1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: nodemanager

Environment

Linux

• Assignee: Nathan Roberts
• Reporter: Nathan Roberts
• Created: 04/Mar/16 19:56
• Updated: 30/Aug/16 01:05
• Resolved: 10/May/16 21:48

Description

Algorithm currently uses "MemFree" + "Inactive" from /proc/meminfo

"Inactive" may not be a very good indication of how much memory can be readily freed because it contains both:

• Pages mapped with MAP_SHARED|MAP_ANONYMOUS (regardless of whether they're being actively accessed or not. Unclear to me why this is the case...)
• Pages mapped MAP_PRIVATE|MAP_ANONYMOUS that have not been accessed recently

Both of these types of pages probably shouldn't be considered "Available".

"Inactive(file)" would seem more accurate but it's not available in all kernel versions. To keep things simple, maybe just use "Inactive(file)" if available, otherwise fallback to "Inactive".

Issue Links

• is required by: YARN-5202 Dynamic Overcommit of Node Resources - POC