[YUNIKORN-2804] [Umbrella] Rethink general retry policy for post allocation failed task

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: core - scheduler, scheduler-interface, shim - kubernetes
• Target Version: 1.7.0
• Assignee: Qi Zhu
• Reporter: Qi Zhu
• Created: 15/Aug/24 03:51
• Updated: 28/Sep/24 00:57

Description

We are adding retry for bind volume failed in:
https://github.com/apache/yunikorn-k8shim/pull/890

Updated, we closed the above PR after discussion.
We need to do the following-up instead of doing the retry above. 

As discussed here, we want to have general retry policy for future improvement.
https://github.com/apache/yunikorn-k8shim/pull/890#issuecomment-2288658926

 

Issue Links

• fixes: YUNIKORN-1596 Pods marked unschedulable when dynamic PVC times out
• incorporates: YUNIKORN-2253 Support retry when bind volume failed case instead of failing the task
• is fixed by: YUNIKORN-2645 Rate limit pod allocations on nodes

Sub-Tasks

• Add troubleshooting docs for bind pod volumes timeout: Resolved
• Create a design doc for general retry policy: In Progress
• Task fail with post allocated but the pod will keep pending: In Progress