[SPARK-6707] Mesos Scheduler should allow the user to specify constraints based on slave attributes

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.5.0
• Component/s: Mesos, Scheduler, Spark Core
• Labels: mesos scheduler
• Target Version/s: 1.5.0
• Assignee: Ankur Chauhan
• Reporter: Ankur Chauhan
• Created: 04/Apr/15 06:34
• Updated: 17/May/20 17:48
• Resolved: 06/Jul/15 23:05

Description

Currently, the mesos scheduler only looks at the 'cpu' and 'mem' resources when trying to determine the usablility of a resource offer from a mesos slave node. It may be preferable for the user to be able to ensure that the spark jobs are only started on a certain set of nodes (based on attributes).

For example, If the user sets a property, let's say

spark.mesos.constraints

is set to

tachyon=true;us-east-1=false

, then the resource offers will be checked to see if they meet both these constraints and only then will be accepted to start new executors.

Issue Links

• links to: [Github] Pull Request #5563 (ankurcha)