[SPARK-8939] YARN EC2 default setting fails with IllegalArgumentException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: EC2
• Assignee: Shivaram Venkataraman
• Reporter: Andrew Or
• Created: 09/Jul/15 03:46
• Updated: 28/Sep/15 12:41
• Resolved: 28/Sep/15 12:41

Description

I just set it up from scratch using the spark-ec2 script. Then I ran

bin/spark-shell --master yarn

which failed with

15/07/09 03:44:29 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Unknown/unsupported param List(--num-executors, , --executor-memory, 6154m, --executor-memory, 6154m, --executor-cores, 2, --name, Spark shell)

This goes away if I provide `--num-executors`, but we should fix the default.