[YARN-8692] Support node utilization metrics for SLS

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: scheduler-load-simulator
• Assignee: Tao Yang
• Reporter: Tao Yang
• Created: 21/Aug/18 10:02
• Updated: 02/Mar/21 14:45

Description

The distribution of node utilization is an important healthy factor for the YARN cluster, related metrics in SLS can be used to evaluate the scheduling effects and optimize related configurations. 

To implement this improvement, we need to do things as below:

(1) Add input configurations (contain avg and stddev for cpu/memory utilization ratio) and generate utilization samples for tasks, not include AM container cause I think it's negligible.

(2) Simulate containers and node utilization within node status. 

(3) calculate and generate the distribution metrics and use standard deviation metric (stddev for short) to evaluate the effects(smaller is better).  

(4) show these metrics on SLS simulator page like this:

For Node memory/CPU utilization distribution graphs, Y-axis is nodes number, and P0 represents 0%~9% utilization ratio(containers-utilization / node-total-resource), P1 represents 10%~19% utilization ratio, P2 represents 20%~29% utilization ratio, ..., at last P9 represents 90%~100% utilization ratio. 

Issue Links

• is duplicated by: YARN-10628 Add node usage metrics in SLS