[YUNIKORN-551] node removal races for lock during scheduling

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.10
• Fix Version/s: 0.10
• Component/s: core - scheduler
• Labels: pull-request-available
• Target Version: 0.10
• Assignee: Wilfred Spiegelenburg
• Reporter: Wilfred Spiegelenburg
• Created: 02/Mar/21 04:12
• Updated: 22/Nov/23 02:31
• Resolved: 03/Mar/21 05:32

Description

A more complicated version of the dead lock mentioned in YUNIKORN-481.

In this case the scheduler is racing with the node removal which in turn removes allocations from the application. The locks taken are al short term locks but it could happen that the application being scheduled also has an allocation on a node being removed.

Scheduling requires the write locked app to request a read lock on the partition to get all known nodes. The partition write locks while removing the node from its internal list and keeps hold of that write lock while removing the allocations which tries to lock the app.

The partition should have released the lock immediately after the node was removed from the list as the rest of the updates are not modifying the partition object.

Issue Links

• causes: YUNIKORN-2171 race between node removal and scheduling cycle
• links to: GitHub Pull Request #253