[YARN-11088] Introduce the config to control the AM allocated to non-exclusive nodes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: capacity scheduler
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Junfan Zhang
• Reporter: Junfan Zhang
• Created: 13/Mar/22 08:50
• Updated: 11/Feb/24 10:35
• Resolved: 31/Mar/22 10:42

Description

Why

Current the implementation of Yarn about AM allocation on non-exclusive nodes is directly to fail fast. I know this aims to keep the stability of job, because the container in non-exclusive nodes will be preempted.

But Yarn cluster in our internal company exists on-premise NodeManagers and elastic NodeManagers (which is built on K8s). When all the elastic nodemanagers decommission, we hope that the AM can be scheduled to non-exclusive nodes.

How to support it

Introduce the new config to control the AM can be allocated to non-exclusive nodes.

Feel free to discuss if having any ideas!

Issue Links

• links to: GitHub Pull Request #4105