Jason Lowe the example provided by Sangjin Lee is the one I wanted to address when I added support for both percentage and absolute cores. Would it make more sense if I picked the lower value instead of one overriding the other. Something like -
1. Evaluate cores allocated by yarn.nodemanager.containers-cpu-cores and yarn.nodemanager.containers-cpu-percentage.
2. Pick the lower of the two values
3. Log a warning/info message that both were specified and that we're picking the lower value.
I'm not thrilled about the name template "containers-cpu-*" since it could easily be misinterpreted as a per-container thing as well, but I'm currently at a loss for a better prefix. Suggestions welcome.
How about yarn.nodemanager.all-containers-cpu-cores and yarn.nodemanager.all-containers-cpu-percentage?
Does getOverallLimits need to check for a quotaUS that's too low as well?
Thanks for catching this; I'll fix it in the next patch.
I think minimally we need to log a warning if we're going to ignore setting up cgroups to limit CPU usage across all containers if the user specified to do so.
I'll add in the logging message.
Related to the previous comment, I think it would be nice if we didn't try to setup any limits if none were specified. That way if there's some issue with correctly determining the number of cores on a particular system it can still work in the default, "use everything" scenario.
NodeManagerHardwareUtils.getContainerCores should be getContainersCores (the per-container vs. all-containers confusion again)
I'll rename the function.