[YUNIKORN-2699] Preemption e2e tests fail in latest master

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Target Version: 1.6.0
• Assignee: Manikandan R
• Reporter: Craig Condit
• Created: 25/Jun/24 18:28
• Updated: 06/Jul/24 07:57
• Resolved: 06/Jul/24 07:57

Description

Output:

 

Preemption Verify_basic_preemption
/home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:139
  STEP: Creating development namespace: dev-anvkm @ 06/25/24 18:08:14.291
  STEP: A queue uses resource more than the guaranteed value even after removing one of the pods. The cluster doesn't have enough resource to deploy a pod in another queue which uses resource less than the guaranteed value. @ 06/25/24 18:08:15.301
  STEP: Update root.sandbox1 and root.sandbox2 with guaranteed memory 4677M @ 06/25/24 18:08:15.301
  STEP: Port-forward the scheduler pod @ 06/25/24 18:08:15.302
port-forward is already running  STEP: Enabling new scheduling config @ 06/25/24 18:08:15.302
  STEP: Deploy the sleep pod sleepjob1 to the development namespace @ 06/25/24 18:08:18.313
  STEP: Deploy the sleep pod sleepjob2 to the development namespace @ 06/25/24 18:08:22.518
  STEP: Deploy the sleep pod sleepjob3 to the development namespace @ 06/25/24 18:08:26.517
  STEP: Deploy the sleep pod sleepjob4 to the development namespace @ 06/25/24 18:08:30.518
  STEP: One of the pods in root.sanbox1 is preempted @ 06/25/24 18:08:38.517
  [FAILED] in [It] - /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:198 @ 06/25/24 18:08:38.718
  Logging yk fullstatedump, spec: Verify_basic_preemption
  Created log file: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/build/e2e/preemption/Verify_basic_preemption_ykFullStateDump.json
  Logging k8s cluster info, spec: Verify_basic_preemption
  Created log file: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/build/e2e/preemption/Verify_basic_preemption_k8sClusterInfo.txt
  Logging yk container logs, spec: Verify_basic_preemption
  Created log file: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/build/e2e/preemption/Verify_basic_preemption_ykContainerLog.txt
  STEP: Tear down namespace: dev-anvkm @ 06/25/24 18:08:39.235
  STEP: Restoring YuniKorn configuration @ 06/25/24 18:08:40.118
  STEP: Restoring the old config maps @ 06/25/24 18:08:40.119
• [FAILED] [27.837 seconds]
Preemption [It] Verify_basic_preemption
/home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:139  [FAILED] One of the pods in root.sandbox1 should be preempted
  Expected
      <int>: 1
  to equal
      <int>: 2
  In [It] at: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:198 @ 06/25/24 18:08:38.718------------------------------ Preemption Verify_preemption_on_priority_queue
/home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:333
  STEP: Creating development namespace: dev-u0kt7 @ 06/25/24 18:10:24.975
  STEP: A task can only preempt a task with lower or equal priority @ 06/25/24 18:10:25.982
  STEP: Update root.sandbox1, root.low-priority, root.high-priority with guaranteed memory 4677M @ 06/25/24 18:10:25.982
  STEP: Port-forward the scheduler pod @ 06/25/24 18:10:25.983
port-forward is already running  STEP: Enabling new scheduling config @ 06/25/24 18:10:25.983
  STEP: Deploy the sleep pod sleepjob1 to the development namespace @ 06/25/24 18:10:28.99
  STEP: Deploy the sleep pod sleepjob2 to the development namespace @ 06/25/24 18:10:32.791
  STEP: Deploy the sleep pod sleepjob3 to the development namespace @ 06/25/24 18:10:35.792
  STEP: Deploy the sleep pod sleepjob4 to the development namespace @ 06/25/24 18:10:38.792
  STEP: Deploy the sleep pod sleepjob5 to the development namespace @ 06/25/24 18:10:38.995
  STEP: The sleep pod sleepjob4 can't be scheduled @ 06/25/24 18:10:39.194
  STEP: The sleep pod sleepjob5 can be scheduled @ 06/25/24 18:10:41.392
  STEP: One of the pods in root.sanbox1 is preempted @ 06/25/24 18:10:46.392
  [FAILED] in [It] - /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:424 @ 06/25/24 18:10:46.592
  Logging yk fullstatedump, spec: Verify_preemption_on_priority_queue
  Created log file: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/build/e2e/preemption/Verify_preemption_on_priority_queue_ykFullStateDump.json
  Logging k8s cluster info, spec: Verify_preemption_on_priority_queue
  Created log file: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/build/e2e/preemption/Verify_preemption_on_priority_queue_k8sClusterInfo.txt
  Logging yk container logs, spec: Verify_preemption_on_priority_queue
  Created log file: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/build/e2e/preemption/Verify_preemption_on_priority_queue_ykContainerLog.txt
  STEP: Tear down namespace: dev-u0kt7 @ 06/25/24 18:10:47.21
  STEP: Restoring YuniKorn configuration @ 06/25/24 18:10:48.393
  STEP: Restoring the old config maps @ 06/25/24 18:10:48.393
• [FAILED] [25.425 seconds]
Preemption [It] Verify_preemption_on_priority_queue
/home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:333  [FAILED] One of the pods in root.sandbox1 should be preempted
  Expected
      <int>: 1
  to equal
      <int>: 2
  In [It] at: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:424 @ 06/25/24 18:10:46.592

Issue Links

• is related to: YUNIKORN-2698 E2e tests for k8shim don't compile with latest core
• relates to: YUNIKORN-2725 Temporarily disable failing e2e preemption tests
• links to: GitHub Pull Request #870
• links to: GitHub Pull Request #909