[YUNIKORN-481] deadlock when removing allocation ask

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.10
• Fix Version/s: 0.10
• Component/s: core - scheduler
• Labels: pull-request-available
• Target Version: 0.10
• Assignee: Wilfred Spiegelenburg
• Reporter: Wilfred Spiegelenburg
• Created: 08/Dec/20 09:14
• Updated: 21/Jan/22 21:48
• Resolved: 10/Dec/20 02:16

Description

A new deadlock was introduced in the scheduler as part of the refactor.

The deadlock can occur when an allocationAsk is removed from an application while the application is scheduled at the same point in time.

The removeAllocationAsk() call in the partitionContext is a write locked call taking the context lock it later tries to take the application write lock. While scheduling to check the nodes we are retrieving the node iterator from the partition. This retrieval is done by a write locked application which needs a read lock on the partition.

The removal of the ask should not hold the write lock on the partition while updating the application. 

Issue Links

• blocks: YUNIKORN-475 Move shim build to core without cache
• is caused by: YUNIKORN-317 Cache/Scheduler modules need to be combined to a single module
• relates to: YUNIKORN-461 Remove allocations map from the partition
• links to: GitHub Pull Request #228