[SPARK-2350] Master throws NPE

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 0.9.2, 1.0.1, 1.1.0
• Component/s: Spark Core
• Assignee: Aaron Davidson
• Reporter: Andrew Or
• Created: 03/Jul/14 00:03
• Updated: 05/Nov/14 10:43
• Resolved: 04/Jul/14 05:32

Description

... if we launch a driver and there are more waiting drivers to be launched. This is because we remove from a list while iterating through this.

Here is the culprit from Master.scala (L487 as of the creation of this JIRA, commit bc7041a42dfa84312492ea8cae6fdeaeac4f6d1c).

for (driver <- waitingDrivers) {
  if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) {
    launchDriver(worker, driver)
    waitingDrivers -= driver
  }
}

Issue Links

• is related to: SPARK-2154 Worker goes down.