[YUNIKORN-1597] Gang scheduling: application might not transition to Running after recovery

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Target Version: 1.3.0
• Assignee: Peter Bacsko
• Reporter: Peter Bacsko
• Created: 23/Feb/23 12:41
• Updated: 20/Jun/23 14:41
• Resolved: 31/Mar/23 19:24

Description

Pods get suck in a certain recovery scenario which involves gang scheduling.

High level overview:
1. All placeholders are running and allocated
2. The real pod is in Pending state
3. Yunikorn crashes and recovers

In this case, the real pod will not transition to Running. It's because:
1. Upon recovery, the state of recovered tasks will be set to "Allocated", not "Bound".
2. If placeholder tasks are already running and allocated, there will be no call to postTaskBound().

A possible fix:
1. In Task.initialize(), set the state to Bound if it's a placeholder task.
2. In Application.onReserving(), check if we have placeholders. If we do, that means we're after recovery, so send an "UpdateReservation" event.

Issue Links

• links to: GitHub Pull Request #541
• links to: GitHub Pull Request #544