I didn't realize about the ContainerStatus being sent to the AM. How would you send the information to the RM then?
We will have to create a new record somewhere in the hierarchy of NodeHeartbeatRequest.
does the need to avoid vcores stem from our intention of moving all the utilization information collection to a separate service that might not be specific to Yarn?
Only partly. The bigger reason is vcore is more of a construct created for scheduling. We can have a vcore-millis which represents more of the resources reserved as opposed to real utilization. Real utilization can be cpu-wall-time, avg cpu utilization etc.
For CPU utilization, the other alternative is to send the number of physical cores on the machine along with percentage usage (like the one top shows).
This reminds me of another thing that I missed mentioning before. We need to send per-container utilization as well as the the host-level utilization.
May be we should first agree here on what exactly we are sending across. The patch should be relatively straigh-forward after that.