[SPARK-34519] ExecutorPodsAllocator use exponential backoff strategy when request executor pod failed

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.0.1
• Component/s: Kubernetes

Environment

spark 3.0.1

kubernetes 1.18.8

• Assignee: Unassigned
• Reporter: Fengyu Cao
• Created: 24/Feb/21 10:27
• Updated: 24/Feb/21 10:27

Description

1. create a resouce quota `kubectl create quota test --hard=cpu=20,memory=60G`
2. submit an application request more than quota `spark-submit --executor-cores 5 --executor-memory 10G --num-executors 10 <spark-pi.py>`
3. seems `ExecutorPodsAllocator: Going to request 5 executors from Kubernetes.` print every second

`spark.kubernetes.allocation.batch.delay` default is 1s, which good enough when allocation succeeded, but exponential backoff maybe an better choice when alloction failed.