[YARN-5724] [Umbrella] Better Queue Management in YARN

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: capacity scheduler
• Assignee: Xuan Gong
• Reporter: Xuan Gong
• Created: 12/Oct/16 19:51
• Updated: 26/Jun/19 15:47

Description

This serves as an umbrella ticket for tasks related to better queue management in YARN.

Today's the only way to manage the queue is through admins editing configuration files and then issuing a refresh command. This will bring many inconveniences. For example, the users can not create / delete /modify their own queues without talking to site level admins.

Even in today's approach (configuration-based), we still have several places needed to improve:

• It is possible today to add or modify queues without restarting the RM, via a CS refresh. But for deleting queue, we have to restart the resourcemanager.
• When a queue is STOPPED, resources allocated to the queue can be handled better. Currently, they'll only be used if the other queues are setup to go over their capacity.

Issue Links

• relates to: YARN-5734 OrgQueue for easy CapacityScheduler queue configuration management
• relates to: YARN-1317 Make Queue, QueueACLs and QueueMetrics first class citizens in YARN

Sub-Tasks

• The state of the parentQueue and its childQueues should be synchronized.: Resolved
• CapacityScheduler: Support deleting queues without requiring a RM restart: Resolved
• Enhancements to STOP queue handling: Resolved
• Add state-machine implementation for scheduler queues: Resolved
• Separate QueueManager from Scheduler: Resolved
• Add documentation for Delete Queue: Resolved
• Add an ability to convert ChildQueue to ParentQueue: Resolved
• make DELETE/STOP/CONVERT queues work in reservation system: Open
• re-visit handling QUEUE-STOP process: Patch Available
• Deleting all children of a Parent Queue on refresh throws exception: Resolved