[REEF-1394] Enable using oversubscribed resources in Mesos clusters

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: REEF-Runtime-Mesos
• Assignee: Unassigned
• Reporter: John Yang
• Created: 18/May/16 07:14
• Updated: 13/Mar/18 13:23

Description

Mesos now allows frameworks to use oversubscribed resources of high-priority user-facing services running in its cluster. (Description: http://mesos.apache.org/documentation/latest/oversubscription/)

It'd be nice to expose this feature in REEF. One possible way to implement this is to attach a tag to EvaluatorRequest for us to specify whether we want regular resources or oversubscribed resources in requesting for evaluators. We can also add a similar tag in EvaluatorDescriptor of AllocatedEvaluator for us to distinguish on which type of resources the allocated evaluator is running. Of course, the default value of the tags should be regular resources for backward compatibility.