[REEF-1753] Intermittent failures of CloseEvaluatorTest in local runtime

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16
• Component/s: REEF Driver, Wake
• Labels: shutdown
• Assignee: Tae-Geon Um
• Reporter: Sergiy Matusevych
• Created: 21/Mar/17 22:34
• Updated: 02/Aug/17 02:15
• Resolved: 27/Mar/17 21:27

Description

mvn test reports:

Failed tests:
  CloseEvaluatorTest.testCloseEvaluator:61 Job state after execution: FORCE_CLOSED

and in the driver logs I see the following:

faulty driver.stderr

Mar 21, 2017 2:06:51 PM org.apache.reef.runtime.common.driver.DriverStatusManager onComplete
INFO: Clean shutdown of the Driver.
Mar 21, 2017 2:06:52 PM org.apache.reef.wake.remote.transport.netty.NettyMessagingTransport open
WARNING: Connection refused. Retry 1 of 3
Mar 21, 2017 2:06:52 PM org.apache.reef.wake.remote.transport.netty.NettyMessagingTransport open
WARNING: Connection refused. Retry 1 of 3
Mar 21, 2017 2:07:01 PM org.apache.reef.wake.remote.impl.RemoteSenderStage close
WARNING: Executor did not terminate in 10,000 ms.
Mar 21, 2017 2:07:01 PM org.apache.reef.wake.remote.transport.netty.NettyMessagingTransport open
WARNING: Thread Thread[org.apache.reef.wake.remote.impl.RemoteSenderStage:pool-06:thread-003,5,main] interrupted while sleeping
Mar 21, 2017 2:07:01 PM org.apache.reef.wake.remote.transport.netty.NettyMessagingTransport open
WARNING: Thread Thread[org.apache.reef.wake.remote.impl.RemoteSenderStage:pool-06:thread-001,5,main] interrupted while sleeping
Mar 21, 2017 2:07:01 PM org.apache.reef.wake.remote.impl.RemoteSenderStage close
WARNING: Executor dropped 0 tasks.
Mar 21, 2017 2:07:04 PM org.apache.reef.runtime.common.driver.DriverRuntimeStopHandler onNext
INFO: Driver shutdown complete

Instead of

normal driver.stderr

Mar 21, 2017 3:18:46 PM org.apache.reef.runtime.common.driver.DriverStatusManager onComplete
INFO: Clean shutdown of the Driver.
Mar 21, 2017 3:18:48 PM org.apache.reef.runtime.common.driver.DriverRuntimeStopHandler onNext
INFO: Driver shutdown complete

P.S. Unfortunately, I am unable to reproduce the error with more granular logging level. If someone can do that, please post your Driver's log here

Issue Links

• is related to: REEF-1729 Fix test job timeouts in Travis CI
• relates to: REEF-1754 Use finer log level when running unit tests in reef-tests