[SPARK-6350] Make mesosExecutorCores configurable in mesos "fine-grained" mode

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.5.0
• Fix Version/s: 1.4.0, 1.5.1, 1.6.0
• Component/s: Mesos
• Target Version/s: 1.4.0, 1.5.1, 1.6.0
• Assignee: Jongyoul Lee
• Reporter: Jongyoul Lee
• Created: 16/Mar/15 07:09
• Updated: 13/Sep/15 10:00
• Resolved: 13/Sep/15 10:00

Description

When spark runs in mesos fine-grained mode, mesos slave launches executor with # of cpus and memories. By the way, # of executor's cores is always CPU_PER_TASKS as same as spark.task.cpus. If I set that values as 5 for running intensive task, mesos executor always consume 5 cores without any running task. This waste resources. We should set executor core as a configuration variable.

Issue Links

• is related to: SPARK-5376 [Mesos] MesosExecutor should have correct resources
• links to: [Github] Pull Request #5063 (jongyoul)
• links to: [Github] Pull Request #8653 (dragos)
• links to: [Github] Pull Request #8732 (dragos)