[YARN-1963] Support priorities across applications within the same queue

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: api, resourcemanager
• Assignee: Sunil G
• Reporter: Arun Murthy
• Created: 19/Apr/14 14:46
• Updated: 14/Oct/19 15:37
• Resolved: 29/Oct/17 05:43

Description

It will be very useful to support priorities among applications within the same queue, particularly in production scenarios. It allows for finer-grained controls without having to force admins to create a multitude of queues, plus allows existing applications to continue using existing queues which are usually part of institutional memory.

Issue Links

• duplicates: MAPREDUCE-4127 Resource manager UI does not show the Job Priority
• is related to: YARN-2099 Preemption in fair scheduler should consider app priorities
• is related to: YARN-2098 App priority support in Fair Scheduler
• is related to: MAPREDUCE-314 Avoid priority inversion that could result due to scheduling running jobs in an order sorted by priority
• is related to: TEZ-2627 Support for Tez Job Priorities
• relates to: MAPREDUCE-5870 Support for passing Job priority through Application Submission Context in Mapreduce Side

Sub-Tasks

• Support for Application priority : Changes in RM and Capacity Scheduler: Resolved
• Priority scheduling support in Capacity scheduler: Resolved
• User level API support for priority label: Resolved
• Priority Label Manager in RM to manage application priority based on configuration: Resolved
• Server side PB changes for Priority Label Manager and Admin CLI support: Resolved
• Support admin cli interface in for Application Priority: Resolved
• FIFO scheduler doesn't respect changing job priority: Resolved
• Support for changing Application priority during runtime: Resolved
• Display Application Priority in RM Web UI: Resolved
• Support for application priority ACLs in queues of CapacityScheduler: Resolved
• REST api support for Application Priority: Resolved
• Support user cli interface in for Application Priority: Resolved
• Publish Application Priority to TimelineServer: Resolved
• Render cluster Max Priority in scheduler metrics in RM web UI: Resolved
• Document ApplicationPriority feature: Resolved
• Runtime Application Priority change should not throw exception for applications at finishing states: Resolved
• Retrospect update ApplicationPriority API return type: Resolved
• AM need to be notified with priority in AllocateResponse: Resolved
• Retrospect app-priority in pendingOrderingPolicy during recovering applications: Resolved
• Pull out priority comparison from fifocomparator and use compound comparator for FifoOrdering policy: Resolved
• updateApplicationPriority api in scheduler should ensure to re-insert app to correct ordering policy: Resolved
• Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611: Resolved