[SPARK-36057] SPIP: Support Customized Kubernetes Schedulers

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Component/s: Kubernetes
• Labels: SPIP
• Assignee: Yikun Jiang
• Reporter: Holden Karau
• Shepherd: Holden Karau
• Created: 08/Jul/21 17:47
• Updated: 15/Mar/23 06:46
• Resolved: 24/Oct/22 19:25

Description

This is an umbrella issue for tracking the work for supporting Volcano & Yunikorn on Kubernetes. These schedulers provide more YARN like features (such as queues and minimum resources before scheduling jobs) that many folks want on Kubernetes.

 

Yunikorn is an ASF project & Volcano is a CNCF project (sig-batch).

 

They've taken slightly different approaches to solving the same problem, but from Spark's point of view we should be able to share much of the code.

 

See the initial brainstorming discussion in SPARK-35623.

 

DISCUSSION: https://lists.apache.org/thread/zv3o62xrob4dvgkbftbv5w5wy75hkbxg

VOTE: https://lists.apache.org/thread/cz3cpp8q4pgmh7h35h6lvkwf6g3lwhcd

VOTE Result: https://lists.apache.org/thread/nvwfo0yo0q8997vs86o7wkjyby4tbp0m

Design DOC: https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg

Recap slide: https://lists.apache.org/thread/mwswfwkycj71npwz8gmv1r5nrvpwj77s

Issue Links

• is related to: SPARK-38396 Improve K8s Integration Tests
• is related to: SPARK-42802 Customized K8s Scheduler GA
• links to: [Github] Pull Request #35015 (Yikun)

Sub-Tasks

• Support replicasets/job API: Resolved
• Add the ability to specify a scheduler: Resolved
• Support for specifiying executor/driver node selector: Resolved
• Add the ability to create resources before driver pod: Resolved
• Add appId interface to KubernetesConf: Resolved
• Add KubernetesCustom[Driver/Executor]FeatureConfigStep developer API: Resolved
• Upgrade kubernetes-client to 5.12.0: Resolved
• Upgrade kubernetes-client to 5.12.2: Resolved
• Add `volcano` module and feature step: Resolved
• Support queue scheduling (Introduce queue) with volcano implementations: Resolved
• Add volcano section to K8s IT README.md: Resolved
• Support priority scheduling with volcano implementations: Resolved
• Bump minimum Volcano version to v1.5.1: Resolved
• Fix Volcano weight to be positive integer and use cpu capability instead: Resolved
• Support APP_ID and EXECUTOR_ID placeholder in annotations: Resolved
• Support driver/executor PodGroup templates: Resolved
• Support resource reservation (Introduce minCPU/minMemory) with volcano implementations: Resolved
• Remove spark.kubernetes.job.queue in favor of spark.kubernetes.driver.podGroupTemplateFile: Resolved
• Set the minimum Volcano version: Resolved
• Remove priorityClassName propagation in favor of explicit settings: Resolved
• Move custom scheduler-specific configs to under `spark.kubernetes.scheduler.NAME` prefix: Resolved
• Unify Statefulset* to StatefulSet*: Resolved
• Add YuniKornSuite: Resolved
• Use YuniKorn v1.1+: Resolved
• Add explicit YuniKorn queue submission test coverage: Resolved
• Add doc for using Apache YuniKorn as a customized scheduler: Resolved
• Volcano feature doesn't work on EKS graviton instances: Resolved
• Volcano queue is not deleted: Resolved
• Introduce `spark.kubernetes.job` sheduling related configurations: Closed
• Support backing off dynamic allocation increases if resources are "stuck": Closed
• [Deprecated] Support the Volcano Job API: Closed
• Check resource after resource creation: Closed
• [CI] Introduce Spark on Kubernetes CI into Volcano community: Closed
• Add doc for "Customized Kubernetes Schedulers": Resolved
• Add doc for Volcano scheduler: Resolved
• Add yunikorn feature step: Resolved
• Support job queue in YuniKorn feature step: Resolved
• Fix doc format/syntax error: Resolved