[YARN-3980] Plumb resource-utilization info in node heartbeat through to the scheduler

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: resourcemanager, scheduler
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Íñigo Goiri
• Reporter: Karthik Kambatla
• Created: 27/Jul/15 15:03
• Updated: 04/Oct/18 14:24
• Resolved: 24/Nov/15 08:29

Description

YARN-1012 and YARN-3534 collect resource utilization information for all containers and the node respectively and send it to the RM on node heartbeat. We should plumb it through to the scheduler so the scheduler can make use of it.

Issue Links

• breaks: YARN-4566 TestMiniYarnClusterNodeUtilization sometimes fails on trunk
• depends upon: YARN-1012 Report NM aggregated container resource utilization in heartbeat
• depends upon: YARN-3534 Collect memory/cpu usage on the node
• depends upon: YARN-4055 Report node resource utilization in heartbeat
• is depended upon by: YARN-1013 CS should watch resource utilization of containers and allocate speculative containers if appropriate
• is depended upon by: YARN-1015 FS should watch node resource utilization and allocate opportunistic containers if appropriate
• is depended upon by: YARN-4292 ResourceUtilization should be a part of NodeInfo REST API
• is related to: YARN-2965 Enhance Node Managers to monitor and report the resource usage on machines
• is related to: HADOOP-12210 Collect network usage on the node
• is related to: HADOOP-12211 Collect disks usages on the node
• is related to: YARN-3481 Report NM aggregated container resource utilization in heartbeat