[YARN-8191] Fair scheduler: queue deletion without RM restart

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.2.0
• Component/s: fairscheduler
• Target Version/s: 3.2.0
• Hadoop Flags: Incompatible change, Reviewed

Release Note

To support Queue deletion with RM restart feature, AllocationFileLoaderService constructor signature was changed. YARN-8390 corrected this and made as a compatible change.

• Assignee: Gergo Repas
• Reporter: Gergo Repas
• Created: 20/Apr/18 13:58
• Updated: 26/Feb/20 05:29
• Resolved: 25/May/18 00:12

Description

The Fair Scheduler never cleans up queues even if they are deleted in the allocation file, or were dynamically created and are never going to be used again. Queues always remain in memory which leads to two following issues.

1. Steady fairshares aren’t calculated correctly due to remaining queues
2. WebUI shows deleted queues, which is confusing for users (YARN-4022).

We want to support proper queue deletion without restarting the Resource Manager:

1. Static queues without any entries that are removed from fair-scheduler.xml should be deleted from memory.
2. Dynamic queues without any entries should be deleted.
3. RM Web UI should only show the queues defined in the scheduler at that point in time.

Issue Links

• causes: YARN-8990 Fix fair scheduler race condition in app submit and queue cleanup
• causes: YARN-8420 Multithreaded correctness Warning occurs in AllocationFileLoaderService
• causes: YARN-8390 Fix API incompatible changes in FairScheduler's AllocationFileLoaderService
• causes: YARN-8994 Fix race condition between move app and queue cleanup in Fair Scheduler
• incorporates: YARN-5399 Add configuration to remember ad-hoc queues upon configuration reload
• is blocked by: YARN-8268 Fair scheduler: reservable queue is configured both as parent and leaf queue
• is duplicated by: YARN-4022 queue not remove from webpage(/cluster/scheduler) when delete queue in xxx-scheduler.xml