[YARN-8896] Limit the maximum number of container assignments per heartbeat

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.0, 3.0.0
• Fix Version/s: 2.10.0, 3.0.4, 3.1.2, 3.2.1
• Assignee: Zhankun Tang
• Reporter: Weiwei Yang
• Created: 17/Oct/18 00:29
• Updated: 21/Sep/20 09:02
• Resolved: 18/Oct/18 19:30

Description

YARN-4161 adds a configuration yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments to control max number of container assignments per heartbeat, however the default value is -1. This could potentially cause the CS gets stuck in the while loop causing issue like YARN-8513. We should change this to a finite number, e.g 100.

Issue Links

• breaks: YARN-8944 TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896
• causes: YARN-8513 CapacityScheduler infinite loop when queue is near fully utilized
• is duplicated by: YARN-10440 resource manager hangs,and i cannot submit any new jobs,but rm and nm processes are normal
• is related to: YARN-8513 CapacityScheduler infinite loop when queue is near fully utilized
• relates to: YARN-8915 Update the doc about the default value of "maximum-container-assignments" for capacity scheduler