[SPARK-27334] Support specify scheduler name for executor pods when submit

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, Spark Core
• Labels: easyfix features
• Assignee: Unassigned
• Reporter: TommyLike
• Created: 01/Apr/19 07:02
• Updated: 17/May/20 18:26
• Resolved: 09/Jun/19 23:15

Description

Currently, there are some external schedulers which bring a lot a great value into kubernetes scheduling especially for HPC case, take a look at the kube-batch (https://github.com/kubernetes-sigs/kube-batch). In order to support it, we had to use Pod Template which seems cumbersome. It would be much convenient if this can be configured via option such as "spark.kubernetes.executor.schedulerName" just like others.

Issue Links

• duplicates: SPARK-24434 Support user-specified driver and executor pod templates