[YUNIKORN-2068] Deadlock when evaluating nodes for preemption

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Target Version: 1.4.0
• Assignee: Craig Condit
• Reporter: Craig Condit
• Created: 23/Oct/23 22:19
• Updated: 17/Jan/24 07:33
• Resolved: 23/Oct/23 23:36

Description

During preemption predicate checks, we have a read lock that can be acquired multiple times without releasing, leading to deadlocks. Problematic ocde is in context.IsPodFitNodeViaPreemption(), where ctx.schedulerCache.GetPod() is called after ctx.schedulerCache.LockForReads(). Both functions acquire the cache rlock.

Issue Links

• is related to: YUNIKORN-2069 Add e2e test covering YUNIKORN-2068
• links to: GitHub Pull Request #697
• links to: GitHub Pull Request #705