[YUNIKORN-2030] Need to check headroom when trying other nodes for reserved allocations

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: core - scheduler
• Labels: pull-request-available
• Target Version: 1.5.0
• Assignee: Yongjun Zhang
• Reporter: Yongjun Zhang
• Created: 11/Oct/23 16:30
• Updated: 20/Mar/24 14:32
• Resolved: 26/Feb/24 10:02

Description

As reported in YUNIKORN-1996, we are seeing many messages like below from time to time:

 WARN    objects/application.go:1504     queue update failed unexpectedly        {“error”: “allocation (map[memory:37580963840 pods:1 vcore:2000]) puts queue ‘root.test-queue’ over maximum allocation (map[memory:3300011278336 vcore:390584]), current usage (map[memory:3291983380480 pods:91 vcore:186000])“}

Restarting Yunikorn helps stoppinging it. Creating this Jira to investigate why it happened, because it's not supposed to happen as we check if there is enough resource headroom before calling 

 

func (sa *Application) tryNode(node *Node, ask *AllocationAsk) *Allocation 

which printed the above message, and only call it when there is enough headroom.

There maybe a bug in headroom checking?

 

Issue Links

• links to: GitHub Pull Request #793