[YUNIKORN-572] terminated app move causes deadlock

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10
• Component/s: core - scheduler
• Labels: pull-request-available
• Target Version: 0.10
• Assignee: Wilfred Spiegelenburg
• Reporter: Wilfred Spiegelenburg
• Created: 15/Mar/21 03:47
• Updated: 21/Jan/22 21:48
• Resolved: 15/Mar/21 21:41

Description

PR #250 for the placeholder cleanup introduced a possible dead lock.

When moving a terminated app from the queue the queue is unlinked from the app. Before that happens we make sure that the queue tracking is up to date. This requires an application lock.

The move used to take a lock on the partition and did all its work in one go. With the change to remove the queue from the app we now lock the app inside the partition lock. Since the app is part of the active list until the move is done scheduling might check the app too. This could lead to a dead lock.

Issue Links

• is related to: YUNIKORN-460 Handle app reservation timeout
• is related to: YUNIKORN-519 Cleanup placeholders when the app is Completed
• links to: GitHub Pull Request #257