[SPARK-44951] Improve Spark Dynamic Allocation

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Kubernetes, Spark Core, YARN
• Assignee: Unassigned
• Reporter: Holden Karau
• Shepherd: Holden Karau
• Created: 24/Aug/23 19:01
• Updated: 08/Mar/24 22:57

Description

For Spark 4 we should aim to improve Spark's dynamic allocation. Some potential ideas here includes the following:

• Plug-gable DEA algorithms
• How to reduce wastage on the RM side? Sometimes the driver asks for some units of resources. But when RM provisions them, the driver cancels it. 
• Support for "warm" executor pools which are not tied to a particular driver but start and wait for a driver to connect to them to "claim" them.
• More explicit Cost Vs AppRunTime confiugration: A good DEA algo should allow the developer to choose between cost and runtime. Sometimes developers might be ok to pay higher costs for faster execution.
• Use previous run information to inform future runs
• Better selection of executors to be scaled down

Sub-Tasks

• Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism: Resolved
• Make DEA algorithms pluggable: Open
• Add the option for dynamically marking containers for preemption based data: Open