[SPARK-1453] Improve the way Spark on Yarn waits for executors before starting

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.0.0
• Component/s: YARN
• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 09/Apr/14 15:17
• Updated: 14/Jul/14 20:32
• Resolved: 14/Jul/14 20:32

Description

Currently Spark on Yarn just delays a few seconds between when the spark context is initialized and when it allows the job to start. If you are on a busy hadoop cluster is might take longer to get the number of executors.

In the very least we could make this timeout a configurable value. Its currently hardcoded to 3 seconds.
Better yet would be to allow user to give a minimum number of executors it wants to wait for, but that looks much more complex.

Issue Links

• duplicates: SPARK-1946 Submit stage after executors have been registered
• is related to: SPARK-1707 Remove 3 second sleep before starting app on YARN
• is related to: SPARK-1946 Submit stage after executors have been registered