[YUNIKORN-1596] Pods marked unschedulable when dynamic PVC times out

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.0
• Assignee: Unassigned
• Reporter: Praveen
• Created: 21/Feb/23 21:37
• Updated: 16/Aug/24 09:13

Description

We are seeing a behavior when a scheduled pod requesting for PVC times out, its marked as unschedulable. There are no retries on such pod and remain in 'pending' state. With pods in pending, autoscaler does not scale down nodes.  This seems similar to issue discussed here:

https://github.com/kubernetes/autoscaler/issues/3409

 

Error from Yunikorn logs :
ERROR cache/context.go:527 Failed to bind pod volumes {"podName": "<PODNAME>", "nodeName": "<IP>", "dynamicProvisions": 1, "staticBindings": 0}
...
...
/workspace/pkg/cache/task.go:382
2023-02-20T00:02:22.368Z ERROR cache/task.go:265 task failed {"appID": "<APPID>", "taskID": "45981d91-e543-459b-9657-bdc03b57e26f", "reason": "bind pod volumes failed, name: <NS/PODNAME>, binding volumes: timed out waiting for the condition”}
{{}}

 

From Autoscalar logs

I0220 20:47:01.775653 1 static_autoscaler.go:502] Scale down status: unneededOnly=true lastScaleUpTime=2023-02-20 19:20:56.429598603 +0000 UTC m=+249612.380355315 lastScaleDownDeleteTime=2023-02-20 06:36:50.929515212 +0000 UTC m=+203766.880271921 lastScaleDownFailTime=2023-02-17 22:01:33.693397034 +0000 UTC m=+49.644153730 scaleDownForbidden=true isDeleteInProgress=false scaleDownInCooldown=true
I0220 20:47:11.787999 1 static_autoscaler.go:228] Starting main loop
I0220 20:47:11.792789 1 filter_out_schedulable.go:65] Filtering out schedulables
I0220 20:47:11.792953 1 scheduler_binder.go:829] All bound volumes for Pod "<podname>" match with Node <node>"
I0220 20:47:11.792981 1 filter_out_schedulable.go:118] Pod <podname> marked as unschedulable can be scheduled on node <node> (based on hinting). Ignoring in scale up.

 

1. Can Yunikorn introduce retries for such scenarios?

1. Can pods be set to error state after retries?

Note: pod name, nodename and ip masked above

Issue Links

• duplicates: YUNIKORN-2253 Support retry when bind volume failed case instead of failing the task
• is fixed by: YUNIKORN-2804 [Umbrella] Rethink general retry policy for post allocation failed task