[YARN-5139] [Umbrella] Move YARN scheduler towards global scheduler

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Wangda Tan
• Reporter: Wangda Tan
• Created: 24/May/16 21:44
• Updated: 21/Sep/19 18:18

Description

Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions because scheduler can only look at one node at the time when scheduling resources.

Pseudo code of existing scheduling logic looks like:

for node in allNodes:
   Go to parentQueue
      Go to leafQueue
        for application in leafQueue.applications:
           for resource-request in application.resource-requests
              try to schedule on node

Considering future complex resource placement requirements, such as node constraints (give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

Issue Links

• is blocked by: YARN-5181 ClusterNodeTracker: add method to get list of nodes matching a specific resourceName
• is duplicated by: YARN-5846 Improve the fairscheduler attemptScheduler
• is related to: YARN-8575 Avoid committing allocation proposal to unavailable nodes in async scheduling
• is required by: YARN-1042 add ability to specify affinity/anti-affinity in container requests
• is required by: YARN-5907 [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests
• relates to: YARN-4902 [Umbrella] Generalized and unified scheduling-strategies in YARN

Sub-Tasks

• Add global scheduler interface definition and update CapacityScheduler to use it.: Resolved
• Update AppSchedulingInfo to use SchedulingPlacementSet: Resolved
• Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes: Resolved
• Update javadocs of new added APIs / classes of scheduler/AppSchedulingInfo: Open
• Should consider utilization of each ResourceType on node while scheduling: Open
• Global scheduler applies to Fair scheduler: Open
• Rename PlacementSet and SchedulingPlacementSet: Resolved
• Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm: Resolved
• Delay scheduling should be an individual policy instead of part of scheduler implementation: Open
• Add multi-node lookup mechanism and pluggable nodes sorting policies to optimize placement decision: Resolved
• Introduce scheduler specific environment variable support in ApplicationSubmissionContext for better scheduling placement configurations: Resolved
• Pending backlog for async allocation threads should be configurable: Resolved
• The capacity scheduler logs too frequently seriously affecting performance: Open
• Resource leak caused by a reserved container being released more than once under async scheduling: Resolved
• Support dynamic policy updates in Capacity Scheduler: Open
• Exclude lagged/unhealthy/decommissioned nodes in async allocating thread: Resolved
• Add multi-thread asynchronous scheduling to fair scheduler: Open
• Use threadPool to handle async scheduling threads: Open
• Skip schedule on not heartbeated nodes in Multi Node Placement: Resolved
• Proactively relocate allocated containers from a stopped node: Open
• Reserved Containers not allocated from available space of other nodes in CandidateNodeSet in MultiNodePlacement: Resolved
• Reserved Containers not allocated from available space of other nodes in CandidateNodeSet in MultiNodePlacement (YARN-10259): Resolved
• Support Multi Node Placement in SingleConstraintAppPlacementAllocator: Resolved
• Import logic of multi-node allocation in CapacityScheduler: Resolved
• Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.: Resolved