[YARN-5035] FairScheduler: Adjust maxAssign dynamically when assignMultiple is turned on

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: fairscheduler
• Target Version/s: 2.9.0
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Introducing a new configuration "yarn.scheduler.fair.dynamic.max.assign" to dynamically determine the resources to assign per heartbeat when assignmultiple is turned on. When turned on, the scheduler allocates roughly half of the remaining resources overriding any max.assign settings configured. This is turned ON by default.

• Assignee: Karthik Kambatla
• Reporter: Karthik Kambatla
• Created: 03/May/16 16:55
• Updated: 28/Oct/16 05:56
• Resolved: 26/May/16 22:20

Description

When assignMultiple is turned on, today the number of containers allocated to node within a single heartbeat is determined by maxAssign. Picking a higher value for this prevents the spreading of the workload, and a smaller number leads to slow ramp up.

The proposal is to make it dynamic - half of the unallocated resources on the node.

Issue Links

• incorporates: YARN-4981 limit the max containers to assign to one application per heartbeart