[SPARK-2945] Allow specifying num of executors in the context configuration

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.3.0
• Component/s: Spark Core, YARN

Environment

Ubuntu precise, on YARN (CDH 5.1.0)

• Target Version/s: 1.3.0
• Assignee: Tao Wang (Inactive)
• Reporter: Shay Rojansky
• Created: 09/Aug/14 08:03
• Updated: 06/Feb/15 19:58
• Resolved: 06/Feb/15 19:58

Description

Running on YARN, the only way to specify the number of executors seems to be on the command line of spark-submit, via the --num-executors switch.

In many cases this is too early. Our Spark app receives some cmdline arguments which determine the amount of work that needs to be done - and that affects the number of executors it ideally requires. Ideally, the Spark context configuration would support specifying this like any other config param.

Our current workaround is a wrapper script that determines how much work is needed, and which itself launches spark-submit with the number passed to --num-executors - it's a shame to have to do this.

Issue Links

• links to: [Github] Pull Request #4335 (WangTaoTheTonic)
• links to: [Github] Pull Request #4350 (WangTaoTheTonic)