[YARN-326] Add multi-resource scheduling to the fair scheduler

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2-alpha
• Fix Version/s: 2.1.0-beta
• Component/s: scheduler
• Labels: incompatible
• Hadoop Flags: Reviewed
• Assignee: Sandy Ryza
• Reporter: Sandy Ryza
• Created: 09/Jan/13 08:10
• Updated: 09/Sep/13 09:42
• Resolved: 03/Jun/13 17:36

Description

With YARN-2 in, the capacity scheduler has the ability to schedule based on multiple resources, using dominant resource fairness. The fair scheduler should be able to do multiple resource scheduling as well, also using dominant resource fairness.

More details to come on how the corner cases with fair scheduler configs such as min and max resources will be handled.

Issue Links

• is blocked by: YARN-2 Enhance CS to schedule accounting for both memory and cpu cores
• is depended upon by: YARN-896 Roll up for long-lived services in YARN
• is depended upon by: YARN-736 Add a multi-resource fair sharing metric
• is duplicated by: YARN-350 FSSchedulerNode is always instantiated with a 0 virtual core capacity
• is duplicated by: YARN-327 Support multiple resources in FS

Sub-Tasks

• Make scheduling mode in FS pluggable: Closed
• FS: Extend SchedulingMode to intermediate queues: Closed
• Refactor fair scheduler to use common Resources: Closed