[YARN-160] nodemanagers should obtain cpu/memory values from underlying OS

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.3-alpha
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: nodemanager
• Labels: BB2015-05-TBR
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Varun Vasudev
• Reporter: Alejandro Abdelnur
• Created: 15/Oct/12 16:37
• Updated: 25/Oct/19 20:26
• Resolved: 26/May/15 18:44

Description

As mentioned in YARN-2

NM memory and CPU configs

Currently these values are coming from the config of the NM, we should be able to obtain those values from the OS (ie, in the case of Linux from /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have an interface that obtains this information. In addition implementations of this interface should be able to specify a mem/cpu offset (amount of mem/cpu not to be avail as YARN resource), this would allow to reserve mem/cpu for the OS and other services outside of YARN containers.

Issue Links

• blocks: YARN-1024 Define a CPU resource(s) unambigiously
• is related to: YARN-1024 Define a CPU resource(s) unambigiously
• relates to: YARN-291 [Umbrella] Dynamic resource configuration
• relates to: YARN-2 Enhance CS to schedule accounting for both memory and cpu cores
• relates to: YARN-3720 Need comprehensive documentation for configuration CPU/memory resources on NodeManager