[YARN-8546] Resource leak caused by a reserved container being released more than once under async scheduling

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0, 3.1.1, 2.10.2
• Component/s: capacity scheduler
• Labels: global-scheduling
• Target Version/s: 3.2.0, 3.1.1
• Hadoop Flags: Reviewed
• Assignee: Tao Yang
• Reporter: Weiwei Yang
• Created: 18/Jul/18 06:55
• Updated: 08/Oct/21 17:30
• Resolved: 08/Oct/21 17:30

Description

I was able to reproduce this issue by starting a job, and this job keeps requesting containers until it uses up cluster available resource. My cluster has 70200 vcores, and each task it applies for 100 vcores, I was expecting total 702 containers can be allocated but eventually there was only 701. The last container could not get allocated because queue used resource is updated to be more than 100%.