[YARN-4963] capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.2, 3.0.0-alpha1
• Fix Version/s: 2.8.0, 3.0.0-alpha2
• Component/s: capacityscheduler
• Labels: oct16-easy
• Hadoop Flags: Reviewed
• Assignee: Nathan Roberts
• Reporter: Nathan Roberts
• Created: 15/Apr/16 15:27
• Updated: 28/Oct/16 18:01
• Resolved: 28/Oct/16 17:45

Description

Currently the capacity scheduler will allow exactly 1 OFF_SWITCH assignment per heartbeat. With more and more non MapReduce workloads coming along, the degree of locality is declining, causing scheduling to be significantly slower. It's still important to limit the number of OFF_SWITCH assignments to avoid densely packing OFF_SWITCH containers onto nodes.

Proposal is to add a simple config that makes the number of OFF_SWITCH assignments configurable.

Will upload candidate patch shortly.

Issue Links

• is required by: YARN-5202 Dynamic Overcommit of Node Resources - POC
• relates to: YARN-5013 Allow applications to provide input on amount of locality delay to use