[SPARK-6640] Executor may connect to HeartbeartReceiver before it's setup in the driver side

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: Spark Core
• Target Version/s: 1.4.0
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 01/Apr/15 02:56
• Updated: 03/Apr/15 18:45
• Resolved: 03/Apr/15 18:45

Description

Here is the current code about starting LocalBackend and creating HeartbeatReceiver:

  // Create and start the scheduler
  private[spark] var (schedulerBackend, taskScheduler) =
    SparkContext.createTaskScheduler(this, master)
  private val heartbeatReceiver = env.actorSystem.actorOf(
    Props(new HeartbeatReceiver(this, taskScheduler)), "HeartbeatReceiver")

When creating LocalBackend, it will start `LocalActor`. `LocalActor` will create Executor, and Executor's constructor will retrieve `HeartbeatReceiver`.

So we should make sure this line:

private val heartbeatReceiver = env.actorSystem.actorOf(
    Props(new HeartbeatReceiver(this, taskScheduler)), "HeartbeatReceiver")

happen before "creating LocalActor".

However, current codes can not guarantee that. Sometimes, creating Executor will crash. The issue was reported by sparkdi <[e-mail]> in http://apache-spark-user-list.1001560.n3.nabble.com/Actor-not-found-td22265.html#a22324

Issue Links

• links to: [Github] Pull Request #5306 (zsxwing)