[YUNIKORN-30] flaky tests cause build failures on PRs

• Type: Test
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.8
• Component/s: test - smoke
• Labels: pull-request-available
• Target Version: 0.8
• Assignee: Wilfred Spiegelenburg
• Reporter: Wilfred Spiegelenburg
• Created: 11/Mar/20 10:25
• Updated: 24/Mar/20 09:24
• Resolved: 24/Mar/20 09:24

Description

Smoke tests have been failing on PR triggered builds.

Failures are inconsistent and linked to multiple test cases, failures in the same tests can even happen in different lines of code in different runs without changes:

2020-03-11T04:39:40.8332236Z --- FAIL: TestSchedulerRecovery (3.07s)
2020-03-11T04:39:40.8340886Z ##[error]    mock_rm_callback.go:175: Failed to wait for allocations, expected 4, actual 3, called from: TestSchedulerRecovery in scheduler_recovery_test.go:213

2020-03-11T04:39:40.9102758Z --- FAIL: TestBasicScheduler (1.11s)
2020-03-11T04:39:40.9103549Z ##[error]    mock_rm_callback.go:175: Failed to wait for allocations, expected 4, actual 3, called from: TestBasicScheduler in scheduler_smoke_test.go:341

2020-03-06T07:17:50.4567697Z --- FAIL: TestReservationForTwoQueues (3.10s)
2020-03-06T07:17:50.4574239Z ##[error]    scheduler_reservation_test.go:276: partition reservations are missing

2020-03-06T08:08:21.8912443Z --- FAIL: TestRemoveReservedNode (1.05s)
2020-03-06T08:08:21.8917559Z ##[error]    scheduler_utils.go:79: Failed to wait for pending resource, expected 80, actual 60, called from: TestRemoveReservedNode in scheduler_reservation_test.go:356

2020-03-04T10:42:16.5788872Z --- FAIL: TestRemoveReservedNode (0.07s)
2020-03-04T10:42:16.5789359Z ##[error]    scheduler_reservation_test.go:357: assertion failed: 2 (int) != 1 (int): reservations missing from app

Issue Links

• links to: GitHub Pull Request #107