[YARN-5764] NUMA awareness support for launching containers

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0, 3.2.0
• Component/s: nodemanager, yarn
• Target Version/s: 3.1.0
• Hadoop Flags: Reviewed
• Assignee: Devaraj Kavali
• Reporter: Olasoji
• Created: 20/Oct/16 22:52
• Updated: 12/Jul/22 16:52
• Resolved: 16/Mar/18 21:59

Description

The purpose of this feature is to improve Hadoop performance by minimizing costly remote memory accesses on non SMP systems. Yarn containers, on launch, will be pinned to a specific NUMA node and all subsequent memory allocations will be served by the same node, reducing remote memory accesses. The current default behavior is to spread memory across all NUMA nodes.

Issue Links

• depends upon: YARN-7033 Add support for NM Recovery of assigned resources (e.g. GPU's, NUMA, FPGA's) to container
• is related to: YARN-11195 Document how to configure NUMA in YARN
• is related to: YARN-11196 NUMA Awareness support in DefaultContainerExecutor
• relates to: YARN-8168 Add support in Winutils for reporting CPU cores in all CPU groups, and aggregate kernel time, idle time and user time for all CPU groups
• requires: YARN-11198 Deletion of assigned resources (e.g. GPU's, NUMA, FPGA's) from State Store