[YARN-6775] CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.1, 3.0.0-alpha3
• Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
• Component/s: capacityscheduler
• Hadoop Flags: Reviewed
• Assignee: Nathan Roberts
• Reporter: Nathan Roberts
• Created: 07/Jul/17 16:42
• Updated: 19/Jul/17 23:26
• Resolved: 19/Jul/17 23:26

Description

There are several things in assignContainers() that are done multiple times even though the result cannot change (canAssignToUser, canAssignToQueue). Add some local caching to take advantage of this fact.

Will post patch shortly. Patch includes a simple throughput test that demonstrates when we have users at their user-limit, the number of NodeUpdateSchedulerEvents we can process can be improved from 13K/sec to 50K/sec.

Issue Links

• is duplicated by: YARN-6243 CapacityScheduler: canAssignToThisQueue should be called only when necessary