[YARN-5139] [Umbrella] Move YARN scheduler towards global scheduler

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Wangda Tan
• Reporter: Wangda Tan
• Created: 24/May/16 21:44
• Updated: 21/Sep/19 18:18

Description

Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions because scheduler can only look at one node at the time when scheduling resources.

Pseudo code of existing scheduling logic looks like:

for node in allNodes:
   Go to parentQueue
      Go to leafQueue
        for application in leafQueue.applications:
           for resource-request in application.resource-requests
              try to schedule on node

Considering future complex resource placement requirements, such as node constraints (give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

Issue Links

• is blocked by: YARN-5181 ClusterNodeTracker: add method to get list of nodes matching a specific resourceName
• is duplicated by: YARN-5846 Improve the fairscheduler attemptScheduler
• is related to: YARN-8575 Avoid committing allocation proposal to unavailable nodes in async scheduling
• is required by: YARN-1042 add ability to specify affinity/anti-affinity in container requests
• is required by: YARN-5907 [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests
• relates to: YARN-4902 [Umbrella] Generalized and unified scheduling-strategies in YARN

Sub-Tasks

• Update javadocs of new added APIs / classes of scheduler/AppSchedulingInfo: Open
• Should consider utilization of each ResourceType on node while scheduling: Open
• Global scheduler applies to Fair scheduler: Open
• Delay scheduling should be an individual policy instead of part of scheduler implementation: Open
• The capacity scheduler logs too frequently seriously affecting performance: Open
• Support dynamic policy updates in Capacity Scheduler: Open
• Add multi-thread asynchronous scheduling to fair scheduler: Open
• Use threadPool to handle async scheduling threads: Open
• Proactively relocate allocated containers from a stopped node: Open