[SPARK-2946] Allow specifying * for --num-executors in YARN

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.0.0
• Component/s: Spark Core

Environment

Ubuntu precise, on YARN (CDH 5.1.0)

• Assignee: Unassigned
• Reporter: Shay Rojansky
• Created: 09/Aug/14 08:07
• Updated: 01/Mar/15 12:06
• Resolved: 01/Mar/15 12:06

Description

It would be useful to allow specifying --num-executors * when submitting jobs to YARN, and to have Spark automatically determine how many total cores are available in the cluster by querying YARN.

Our scenario is multiple users running research batch jobs. We never want to have a situation where cluster resources aren't being used, so ideally users would specify * and let YARN scheduling and preemption ensure fairness.

Issue Links

• duplicates: SPARK-3183 Add option for requesting full YARN cluster