[YUNIKORN-1526] support K8s pod overhead

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Target Version: 1.3.0
• Assignee: Qi Zhu
• Reporter: Wilfred Spiegelenburg
• Created: 02/Jan/23 23:56
• Updated: 20/Jun/23 14:41
• Resolved: 13/Feb/23 22:20

Description

With K8s 1.24 a pod can now provide an overhead in the pod spec: pod.Spec.Overhead. The pod overhead allows specifying an overhead based on the runtime set on the pod. For certain runtimes this overhead can be large.  See this KEP for details.

The scheduler should take into account this overhead if set on a pod. We currently calculate the size of the pod based on the containers but do not take the overhead into account. That needs to change.

We need to take the overhead into account as part of scheduling and quota calculations: include the pod.Spec.Overhead resources in the size of the pod before sending it to the core. Overhead only support cpu and memory and is added to the requests. The plugin framework (node related checks) calculates the pod size each time it is called (overhead!) and includes the overhead. The callback for the predicates must take that into account and not be broken by that implementation.

Adding clearly how we have calculated the overall pod size in the logging is a requirement.

Issue Links

• split to: YUNIKORN-1559 rethink QOS BestEffort pod scheduling
• links to: GitHub Pull Request #520