[SPARK-2949] SparkContext does not fate-share with ActorSystem

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Aaron Davidson
• Created: 09/Aug/14 21:12
• Updated: 23/Jan/16 23:14
• Resolved: 23/Jan/16 23:14

Description

It appears that an uncaught fatal error in Spark's Driver ActorSystem does not cause the SparkContext to terminate. We observed an issue in production that caused a PermGen error, but it just kept throwing this error:

14/08/09 15:07:24 ERROR ActorSystemImpl: Uncaught fatal error from thread [spark-akka.actor.default-dispatcher-26] shutting down ActorSystem [spark]
java.lang.OutOfMemoryError: PermGen space

We should probably do something similar for what we did in the DAGSCheduler and ensure that we call SparkContext#stop() if the entire ActorSystem dies with a fatal error.

Issue Links

• relates to: SPARK-5293 Enable Spark user applications to use different versions of Akka