[SPARK-1620] Uncaught exception from Akka scheduler

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.9.0, 1.0.0
• Fix Version/s: 1.0.0
• Component/s: Spark Core
• Assignee: Mark Hamstra
• Reporter: Mark Hamstra
• Created: 24/Apr/14 22:49
• Updated: 14/May/14 17:07
• Resolved: 14/May/14 17:07

Description

I've been looking at this one in the context of a BlockManagerMaster that OOMs and doesn't respond to heartBeat(), but I suspect that there may be problems elsewhere where we use Akka's scheduler.

The basic nature of the problem is that we are expecting exceptions thrown from a scheduled function to be caught in the thread where ActorSystem.scheduler.schedule() or scheduleOnce() has been called. In fact, the scheduled function runs on its own thread, so any exceptions that it throws are not caught in the thread that called schedule() – e.g., unanswered BlockManager heartBeats (scheduled in BlockManager#initialize) that end up throwing exceptions in BlockManagerMaster#askDriverWithReply do not cause those exceptions to be handled by the Executor thread's UncaughtExceptionHandler.