[YARN-4066] Large number of queues choke fair scheduler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: fairscheduler
• Hadoop Flags: Reviewed
• Flags: Patch
• Assignee: Johan Gustavsson
• Reporter: Johan Gustavsson
• Created: 20/Aug/15 05:27
• Updated: 30/Aug/16 01:17
• Resolved: 29/Sep/15 15:16

Description

Due to synchronization and all the loops performed during queue creation, setting a large amount of queues (12000+) will completely choke the scheduler. To deal with this some optimization to "QueueManager.updateAllocationConfiguration(AllocationConfiguration queueConf)" should be done to reduce the amount of unnesecary loops. The attached patch have been tested to work with atleast 96000 queues.