[REEF-291] Sporadic job failures in REEF-Tests

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16
• Component/s: REEF-Tests
• Assignee: Brian Cho
• Reporter: Brian Cho
• Created: 29/Apr/15 14:41
• Updated: 15/Nov/16 18:27
• Resolved: 15/Nov/16 18:27

Description

We've been seeing sporadic reef-tests failures in the Windows CI, e.g. Windows-265. It appears jobs that should fail right away are instead reaching the one-minute timeout. I am now also seeing this semi-regularly on a Ubuntu box.

This:

Apr 27, 2015 11:05:31 AM org.apache.reef.client.DriverLauncher run
WARNING: The Job timed out.
Apr 27, 2015 11:05:31 AM org.apache.reef.runtime.common.client.RunningJobsImpl closeAllJobs
WARNING: Force close job FailTaskClose

Leads to this:

Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 85.547 sec <<< FAILURE! - in org.apache.reef.tests.fail.FailTaskTest
testFailTaskClose(org.apache.reef.tests.fail.FailTaskTest)  Time elapsed: 60.183 sec  <<< FAILURE!
java.lang.AssertionError: expected:<FAILED> but was:<FORCE_CLOSED>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.apache.reef.tests.TestUtils.assertLauncherFailure(TestUtils.java:39)
	at org.apache.reef.tests.fail.FailTaskTest.failOn(FailTaskTest.java:52)
	at org.apache.reef.tests.fail.FailTaskTest.testFailTaskClose(FailTaskTest.java:91)

Issue Links

• is superceded by: REEF-1668 Intermittent failures of EvaluatorCloseTest

Sub-Tasks

• Allow clean multiple calls to Wake Clock close: Resolved
• Avoid holding Wake schedule lock while calling event handlers: Resolved
• Timeout on Fail_AllocatedEvaluator: Resolved
• Fail_Alarm fails with assumed state RUNNING but reported state FAILED: Resolved
• Fix a bug in WatcherTest: Resolved