[YUNIKORN-2866] [UMBRELLA] Support InPlacePodVerticalScaling (phase 2)

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: core - scheduler, shim - kubernetes
• Target Version: 1.7.0
• Assignee: Craig Condit
• Reporter: Craig Condit
• Created: 09/Sep/24 19:12
• Updated: 16/Oct/24 16:45

Description

Kubernetes 1.27 added a new InPlacePodVerticalScaling feature. While this is currently still in an alpha state as of 1.30 (and therefore requires a feature flag to enable), it will likely be moved to beta in 1.32, meaning it will be enabled by default, and considered stable in an upcoming release. The implementation of this feature has implications for YuniKorn, as with the feature enabled, the requests and limits of a Pod are no longer immutable.

Fortunately, the updated API objects that enable the feature contain the new fields so we can add initial support for the feature now. To enable the feature for testing in a Kind cluster, the kind cluster configuration needs to contain the following:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  "InPlacePodVerticalScaling": true

During scheduling of new pods, the requested resources are still used as before.

However, once a pod has been started, the actual resource utilization needs to be tracked via a new Pod.Status.ContainerStatuses[].AllocatedResources collection. In addition, if the value of Pod.Status.Resize is set to Proposed, the usage of each container needs to be computed as the maximum of its requested and allocated resources. The requested resources field becomes mutable once this feature is turned on, and it represents the latest requested (not actual) usage of the container.

Supporting this feature is not optional within YuniKorn, as failure to process the updated resources will mean that we do not account for resource usage correctly if a pod is updated.

Several steps will need to be taken to support this properly:

• Ensure that GetPodResources() accurately computes the effective usage of the Pod in all cases. Since the AllocatedResources field will not be populated when this feature is not active, and is only set once the pod is in a running statue, this is fairly straightforward and can be implemented even in clusters which do not have this feature enabled.
• The result of GetPodResources() will need to be cached in the shim so that we can detect resource changes on Pod updates. Comparing the result of GetPodResources() on the new Pod vs. the existing version will allow us to easily detect changes.
• If changes are detected to a running YuniKorn-managed pod, an update message will need to be sent from the core to change the resources of the allocated task. 
• If changes are detected to a running non-Yunikorn-managed pod, and update of the node utilized resources will need to be sent from the shim to the core.
• The core must not reject these updates, even if they would cause a queue to go over capacity. Instead, they must be applied to the appropriate ask or allocation unconditionally.

Issue Links

• is duplicated by: YUNIKORN-1765 [Umbrella] Support K8s vertical pod auto scaling
• relates to: YUNIKORN-2791 [Umbrella] Tracking non-Yunikorn allocations in the core
• split from: YUNIKORN-2365 [UMBRELLA] Support InPlacePodVerticalScaling (phase 1)

Sub-Tasks

• Add e2e tests for InPlacePodVerticalScaling feature: In Progress
• Validate InPlacePodVerticalScaling feature once it enters beta: Open
• Update MockScheduler test case with foreign pod resource update: Resolved
• [core] Update foreign pod resource usage: Resolved