[YARN-3306] [Umbrella] Proposing per-queue Policy driven scheduling in YARN

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: scheduler
• Assignee: Vinod Kumar Vavilapalli
• Reporter: Vinod Kumar Vavilapalli
• Created: 09/Mar/15 17:53
• Updated: 07/Jan/17 01:52

Description

Scheduling layout in Apache Hadoop YARN today is very coarse grained. This proposal aims at converting today’s rigid scheduling in YARN to a per­-queue policy driven architecture.

We propose the creation of a c​ommon policy framework​ and implement a​common set of policies​ that administrators can pick and chose per queue

• Make scheduling policies configurable per queue
• Initially, we limit ourselves to a new type of scheduling policy that determines the ordering of applications within the l​eaf ­queue
• In the near future, we will also pursue parent­ queue level policies and potential algorithm reuse through a separate type of policies that control resource limits per queue, user, application etc.

Sub-Tasks

• Create Initial OrderingPolicy Framework and FifoOrderingPolicy: Resolved
• Implement a FairOrderingPolicy: Resolved
• Support a Priority OrderingPolicy: Open
• Introduce the notion of policies for a parent queue: Open
• Consider abstracting out user, app limits etc into some sort of a LimitPolicy: Open
• Integrate OrderingPolicy Framework with CapacityScheduler: Resolved
• Create an extension of ProportionalCapacityPreemptionPolicy which preempts a number of containers from each application in a way which respects fairness: Resolved
• Ordering policy should be allowed to reorder an application when demand changes: Resolved
• pendingApplications in LeafQueue should also use OrderingPolicy: Resolved