[YUNIKORN-2665] Gang app originator pod changes after restart

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.0, 1.4.0, 1.5.0, 1.5.1
• Fix Version/s: 1.5.2, 1.6.0
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Target Version: 1.5.2, 1.6.0
• Assignee: Manikandan R
• Reporter: Manikandan R
• Created: 05/Jun/24 11:46
• Updated: 13/Jun/24 01:28
• Resolved: 13/Jun/24 01:28

Description

Gang app choose the first pod (who created the app) as originator pod which becomes the real driver pod later. While processing gang app specifically after the placeholder creation and in the process of replacement, restart can lead to the below described incorrect behaviour:

During restore, there is no guarantee on the ordering of pods coming from K8s lister especially when all the pods created with the same second timestamp. k8s use the seconds based timestamp, which means all pods created with in same second has same timestamp. During this situation, whichever pod comes first from lister, YK designate it as originator pod. So, any placeholder could become the originator pod and actual originator pod has been lost. This change could cause rippling effects leading to weird behaviour and needs to be fixed.

Issue Links

• relates to: YUNIKORN-2667 E2E test for Gang app originator pod changes after restart
• links to: GitHub Pull Request #855