[YUNIKORN-2808] E2E test Verify_preemption_on_priority_queue test is flaky

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Target Version: 1.6.0
• Assignee: Manikandan R
• Reporter: Craig Condit
• Created: 15/Aug/24 23:05
• Updated: 28/Aug/24 15:52
• Resolved: 28/Aug/24 15:52

Description

Latest core from master when integrated with shim causes e2e tests to fail sporadically:

Preemption Verify_preemption_on_priority_queue
1587/home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:333
1588  STEP: Creating development namespace: dev-89y29 @ 08/15/24 22:22:43.966
1589  STEP: A task can only preempt a task with lower or equal priority @ 08/15/24 22:22:44.973
1590  STEP: Update root.sandbox1, root.low-priority, root.high-priority with guaranteed memory 4459M @ 08/15/24 22:22:44.973
1591  STEP: Port-forward the scheduler pod @ 08/15/24 22:22:44.973
1592port-forward is already running  STEP: Enabling new scheduling config @ 08/15/24 22:22:44.974
1593  STEP: Deploy the sleep pod sleepjob1 to the development namespace @ 08/15/24 22:22:47.982
1594  STEP: Deploy the sleep pod sleepjob2 to the development namespace @ 08/15/24 22:22:51.786
1595  STEP: Deploy the sleep pod sleepjob3 to the development namespace @ 08/15/24 22:22:55.786
1596  STEP: Deploy the sleep pod sleepjob4 to the development namespace @ 08/15/24 22:22:59.785
1597  STEP: Deploy the sleep pod sleepjob5 to the development namespace @ 08/15/24 22:22:59.989
1598  STEP: The sleep pod sleepjob4 can't be scheduled @ 08/15/24 22:23:00.188
1599  STEP: The sleep pod sleepjob5 can be scheduled @ 08/15/24 22:23:02.385
1600  STEP: One of the pods in root.sanbox1 is preempted @ 08/15/24 22:23:26.786
1601  [FAILED] in [It] - /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:424 @ 08/15/24 22:23:26.986
1602  Logging yk fullstatedump, spec: Verify_preemption_on_priority_queue
1603  Created log file: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/build/e2e/preemption/Verify_preemption_on_priority_queue_ykFullStateDump.json
1604  Logging k8s cluster info, spec: Verify_preemption_on_priority_queue
1605  Created log file: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/build/e2e/preemption/Verify_preemption_on_priority_queue_k8sClusterInfo.txt
1606  Logging yk container logs, spec: Verify_preemption_on_priority_queue
1607  Created log file: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/build/e2e/preemption/Verify_preemption_on_priority_queue_ykContainerLog.txt
1608  STEP: Tear down namespace: dev-89y29 @ 08/15/24 22:23:27.441
1609  STEP: Restoring YuniKorn configuration @ 08/15/24 22:23:29.585
1610  STEP: Restoring the old config maps @ 08/15/24 22:23:29.586
1611• [FAILED] [47.626 seconds]
1612Preemption [It] Verify_preemption_on_priority_queue
1613/home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:333
1614
1615  [FAILED] One of the pods in root.sandbox1 should be preempted
1616  Expected
1617      <int>: 1
1618  to equal
1619      <int>: 2
1620  In [It] at: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:424 @ 08/15/24 22:23:26.986

I believe this was caused by YUNIKORN-2769. It's not clear whether the test or core logic is in error.

Issue Links

• is caused by: YUNIKORN-2769 Preemption fails between two siblings when preemptor is UG and parent is above OG
• links to: GitHub Pull Request #897
• links to: GitHub Pull Request #955