[SPARK-1235] DAGScheduler ignores exceptions thrown in handleTaskCompletion

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.9.0, 0.9.1
• Fix Version/s: 1.0.0
• Assignee: Nan Zhu
• Reporter: Kay Ousterhout
• Created: 12/Mar/14 13:05
• Updated: 25/Apr/14 23:06
• Resolved: 25/Apr/14 23:06

Description

If an exception gets thrown in the handleTaskCompletion method, the method exits, but the exception is caught somewhere (not clear where) and the DAGScheduler keeps running. Jobs hang as a result – because not all of the task completion code gets run.

This was first reported by Brad Miller on the mailing list: http://apache-spark-user-list.1001560.n3.nabble.com/Fwd-pyspark-crash-on-mesos-td2256.html and this behavior seems to have changed since 0.8 (when, based on Brad's description, it sounds like an exception in handleTaskCompletion would cause the DAGScheduler to crash), suggesting that this may be related to the Scala 2.10.3.

To reproduce this problem, add "throw new Exception("foo")" anywhere in handleTaskCompletion and run any job locally. The job will hang and you can see the exception get printed in the logs.