[SPARK-42766] YarnAllocator should filter excluded nodes when launching allocated containers

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.2
• Component/s: YARN
• Assignee: Unassigned
• Reporter: wangshengjie
• Created: 13/Mar/23 03:38
• Updated: 13/Mar/23 06:32

Description

In production environment, we hit an issue like this:

If we request 10 containers form nodeA and nodeB, first response from Yarn return 5 contianers from nodeA and nodeB, then nodeA blacklisted, and second response from Yarn maybe return some containers from nodeA and launching containers, but when containers(Executor) setup and send register request to Driver, it will be rejected and this failure will be counted to 

spark.yarn.max.executor.failures 

, and will casue app failed.

Max number of executor failures ($maxNumExecutorFailures) reached

 

Issue Links

• links to: [Github] Pull Request #40391 (wangshengjie123)