[SPARK-2635] Fix race condition at SchedulerBackend.isReady in standalone mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: Spark Core
• Assignee: Zhihui
• Reporter: Zhihui
• Created: 23/Jul/14 02:48
• Updated: 09/Aug/14 05:55
• Resolved: 09/Aug/14 05:54

Description

In SPARK-1946(PR #900), configuration spark.scheduler.minRegisteredExecutorsRatio was introduced. However, in standalone mode, there is a race condition where isReady() can return true because totalExpectedExecutors has not been correctly set.

Because expected executors is uncertain in standalone mode, the PR try to use CPU cores(--total-executor-cores) as expected resources to judge whether SchedulerBackend is ready.

Issue Links

• is related to: SPARK-1946 Submit stage after executors have been registered