Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.0.1
-
None
-
None
-
spark 3.0.1
kubernetes 1.18.8
Description
- create a resouce quota `kubectl create quota test --hard=cpu=20,memory=60G`
- submit an application request more than quota `spark-submit --executor-cores 5 --executor-memory 10G --num-executors 10 <spark-pi.py>`
- seems `ExecutorPodsAllocator: Going to request 5 executors from Kubernetes.` print every second
`spark.kubernetes.allocation.batch.delay` default is 1s, which good enough when allocation succeeded, but exponential backoff maybe an better choice when alloction failed.