[SPARK-6050] Spark on YARN does not work --executor-cores is specified

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.0
• Component/s: YARN

Environment

2.5 based YARN cluster.

• Target Version/s: 1.3.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Mridul Muralidharan
• Created: 27/Feb/15 03:38
• Updated: 09/Mar/15 18:57
• Resolved: 02/Mar/15 22:42

Description

There are multiple issues here (which I will detail as comments), but to reproduce running the following ALWAYS hangs in our cluster with the 1.3 RC

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --executor-cores 8 --num-executors 15 --driver-memory 4g --executor-memory 2g --queue webmap lib/spark-examples*.jar 10

Issue Links

• links to: [Github] Pull Request #4818 (vanzin)