[SPARK-24942] Improve cluster resource management with jobs containing barrier stage

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Spark Core
• Target Version/s: 3.2.0
• Epic Link: Support Barrier Execution Mode
• Assignee: Unassigned
• Reporter: Xingbo Jiang
• Created: 27/Jul/18 03:45
• Updated: 06/Feb/23 19:30

Description

https://github.com/apache/spark/pull/21758#discussion_r205652317

We shall improve cluster resource management to address the following issues:

• With dynamic resource allocation enabled, it may happen that we acquire some executors (but not enough to launch all the tasks in a barrier stage) and later release them due to executor idle time expire, and then acquire again.
• There can be deadlock with two concurrent applications. Each application may acquire some resources, but not enough to launch all the tasks in a barrier stage. And after hitting the idle timeout and releasing them, they may acquire resources again, but just continually trade resources between each other.