[SPARK-39025] Add k8s Scheduling, Preemption and Eviction feature to apache spark

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.2.1
• Component/s: Kubernetes
• Assignee: Unassigned
• Reporter: Qian Sun
• Created: 26/Apr/22 10:26
• Updated: 12/Dec/22 18:11
• Resolved: 08/May/22 01:23

Description

As https://kubernetes.io/docs/concepts/scheduling-eviction/ show, apache spark lacks of support for k8s scheduling, preemption and eviction.

This issue wants to support toleration/priorityClass/runtimeClass etc , more information please refer 

https://kubernetes.io/docs/concepts/scheduling-eviction/ 

https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/

https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/

https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/

https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/

https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/

https://kubernetes.io/docs/concepts/scheduling-eviction/resource-bin-packing/

https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/

https://kubernetes.io/docs/concepts/scheduling-eviction/scheduler-perf-tuning/

Issue Links

• duplicates: SPARK-24434 Support user-specified driver and executor pod templates
• links to: [Github] Pull Request #36360 (dcoliversun)
• links to: [Github] Pull Request #36360 (dcoliversun)

Sub-Tasks

• Add k8s tolerations support for apache spark: Resolved