[SPARK-4922] Support dynamic allocation for coarse-grained Mesos

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.2.0
• Component/s: Mesos
• Assignee: Unassigned
• Reporter: Andrew Or
• Created: 22/Dec/14 18:57
• Updated: 20/Jul/15 08:15
• Resolved: 20/Jul/15 08:15

Description

This brings SPARK-3174, which provided dynamic allocation of cluster resources to Spark on YARN applications, to Mesos coarse-grained mode.

Note that the translation is not as trivial as adding a code path that exposes the request and kill mechanisms as we did for YARN is SPARK-3822. This is because Mesos coarse-grained mode schedules on the notion of setting the number of cores allowed for an application (as in standalone mode) instead of number of executors (as in YARN mode). For more detail, please see SPARK-4751.

If you intend to work on this, please provide a detailed design doc!

Issue Links

• is duplicated by: SPARK-6287 Add support for dynamic allocation in the Mesos coarse-grained scheduler
• is related to: SPARK-4751 Support dynamic allocation for standalone mode
• is related to: SPARK-1882 Support dynamic memory sharing in Mesos
• is related to: SPARK-3174 Provide elastic scaling within a Spark application
• is related to: SPARK-5349 Spark standalone should support dynamic resource scaling