[SPARK-36057] SPIP: Support Customized Kubernetes Schedulers

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Component/s: Kubernetes
• Labels: SPIP
• Assignee: Yikun Jiang
• Reporter: Holden Karau
• Shepherd: Holden Karau
• Created: 08/Jul/21 17:47
• Updated: 15/Mar/23 06:46
• Resolved: 24/Oct/22 19:25

Description

This is an umbrella issue for tracking the work for supporting Volcano & Yunikorn on Kubernetes. These schedulers provide more YARN like features (such as queues and minimum resources before scheduling jobs) that many folks want on Kubernetes.

 

Yunikorn is an ASF project & Volcano is a CNCF project (sig-batch).

 

They've taken slightly different approaches to solving the same problem, but from Spark's point of view we should be able to share much of the code.

 

See the initial brainstorming discussion in SPARK-35623.

 

DISCUSSION: https://lists.apache.org/thread/zv3o62xrob4dvgkbftbv5w5wy75hkbxg

VOTE: https://lists.apache.org/thread/cz3cpp8q4pgmh7h35h6lvkwf6g3lwhcd

VOTE Result: https://lists.apache.org/thread/nvwfo0yo0q8997vs86o7wkjyby4tbp0m

Design DOC: https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg

Recap slide: https://lists.apache.org/thread/mwswfwkycj71npwz8gmv1r5nrvpwj77s

Issue Links

• is related to: SPARK-38396 Improve K8s Integration Tests
• is related to: SPARK-42802 Customized K8s Scheduler GA
• links to: [Github] Pull Request #35015 (Yikun)