[YARN-3332] [Umbrella] Unified Resource Statistics Collection per node

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Vinod Kumar Vavilapalli
• Reporter: Vinod Kumar Vavilapalli
• Created: 10/Mar/15 19:56
• Updated: 08/Oct/17 18:09

Description

Today in YARN, NodeManager collects statistics like per container resource usage and overall physical resources available on the machine. Currently this is used internally in YARN by the NodeManager for only a limited usage: automatically determining the capacity of resources on node and enforcing memory usage to what is reserved per container.

This proposal is to extend the existing architecture and collect statistics for usage b​eyond​ the existing use­cases.

Proposal attached in comments.

Issue Links

• is depended upon by: YARN-2745 Extend YARN to support multi-resource packing of tasks
• is related to: YARN-3334 [Event Producers] NM TimelineClient container metrics posting to new timeline service.
• relates to: YARN-2928 YARN Timeline Service v.2: alpha 1
• relates to: YARN-2745 Extend YARN to support multi-resource packing of tasks
• relates to: YARN-2141 [Umbrella] Capture container and node resource consumption