[SPARK-5095] Support launching multiple mesos executors in coarse grained mesos mode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 2.0.0
• Component/s: Mesos
• Target Version/s: 2.0.0
• Assignee: Timothy Chen
• Reporter: Timothy Chen
• Created: 06/Jan/15 00:09
• Updated: 12/Feb/16 20:29
• Resolved: 10/Feb/16 18:53

Description

Currently in coarse grained mesos mode, it's expected that we only launch one Mesos executor that launches one JVM process to launch multiple spark executors.

However, this become a problem when the JVM process launched is larger than an ideal size (30gb is recommended value from databricks), which causes GC problems reported on the mailing list.

We should support launching mulitple executors when large enough resources are available for spark to use, and these resources are still under the configured limit.

This is also applicable when users want to specifiy number of executors to be launched on each node

Issue Links

• is related to: SPARK-12248 Make Spark Coarse Mesos Scheduler obey limits on memory/cpu ratios
• links to: [Github] Pull Request #4027 (tnachen)
• links to: [Github] Pull Request #10993 (mgummelt)
• links to: [Github] Pull Request #11164 (mgummelt)
• links to: [Github] Pull Request #11187 (mgummelt)