[YUNIKORN-2737] Cleanup handleFailApplicationEvent handling

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: shim - kubernetes
• Assignee: Tzu-Hua Lan
• Reporter: Wilfred Spiegelenburg
• Created: 11/Jul/24 05:57
• Updated: 26/Aug/24 02:37

Description

When we handle a failed application in the shim in handleFailApplicationEvent() we call the placeholder cleanup.
Three issues:

• The cleanup needs the app lock after it takes the mgr lock. The app lock is already held when we process the event. Should be placing the cleanup last to not hold the manager lock for longer than needed
• failing an application is triggered by the core which should do the cleanup already so this might be redundant to start with.
• The failure handling also marks unassigned pods as failed which means there is an overlap between the failure handling and the placeholder cleanup which we should remove. Either ignore all placeholders in the failure or only cleanup assigned placeholders.