[YUNIKORN-1933] Verify_allow_preemption_tag e2e test is flaky

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Target Version: 1.4.0
• Assignee: Hsuan Zong Wu
• Reporter: Craig Condit
• Created: 24/Aug/23 20:12
• Updated: 20/Nov/23 16:42
• Resolved: 31/Aug/23 17:32

Description

The Verify_allow_preemption_tag test added in YUNIKORN-1877 is flaky and has been failing on unrelated PRs:

Preemption Verify_allow_preemption_tag
/home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:433
  STEP: The value of 'false' for the allow preemption annotation on the PriorityClass moves the Pod to the back of the preemption list @ 08/24/23 19:24:28.564
  STEP: Update root.sandbox1, root.sandbox2 and root.sandbox3 with guaranteed memory 1509M @ 08/24/23 19:24:28.564
  STEP: Port-forward the scheduler pod @ 08/24/23 19:24:28.564
port-forward is already running  STEP: Enabling new scheduling config @ 08/24/23 19:24:28.564
  STEP: Creating priority class allow-preemption @ 08/24/23 19:24:29.579
  STEP: Creating priority class preemption-not-allow @ 08/24/23 19:24:29.584
  STEP: Deploy the sleep pod sleepjob1 to the development namespace @ 08/24/23 19:24:29.588
  STEP: Deploy the sleep pod sleepjob2 to the development namespace @ 08/24/23 19:24:32.947
  STEP: Deploy the sleep pod sleepjob3 to the development namespace @ 08/24/23 19:24:36.947
  [FAILED] in [It] - /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:517 @ 08/24/23 19:25:38.347
  STEP: Delete all sleep pods @ 08/24/23 19:25:38.347
  STEP: Restoring YuniKorn configuration @ 08/24/23 19:25:39.745
  STEP: Restoring the old config maps @ 08/24/23 19:25:39.746
• [FAILED] [73.195 seconds]
Preemption [It] Verify_allow_preemption_tag
/home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:433

  [FAILED] Unexpected error:
      <wait.errInterrupted>: 
      timed out waiting for the condition
      {
          cause: <*errors.errorString | 0xc00036bff0>{
              s: "timed out waiting for the condition",
          },
      }
  occurred
  In [It] at: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/preemption/preemption_test.go:517 @ 08/24/23 19:25:38.347
------------------------------

Issue Links

• links to: GitHub Pull Request #663