[SPARK-42802] Customized K8s Scheduler GA

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Kubernetes
• Labels: releasenotes
• Target Version/s: 3.4.0
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 15/Mar/23 06:44
• Updated: 15/Mar/23 06:54
• Resolved: 15/Mar/23 06:54

Description

Since Apache Spark 3.3.1, we have been supported the customized K8s scheduler via SPARK-36057. This issue aims to announce GA of the customized K8s schedulers in Apache Spark 3.4.0.

Issue Links

• relates to: SPARK-36057 SPIP: Support Customized Kubernetes Schedulers

Sub-Tasks

• Update YuniKorn documentation with v1.2: Resolved
• Remove ARM limitation for YuniKorn from docs: Resolved
• Upgrade Volcano to 1.7.0: Resolved
• Remove `experimental` from Volcano docs: Resolved
• Enable Volcano unit tests and integration tests in GitHub Action: Resolved
• Make K8s volcano IT work in Github Action: Resolved
• Use Java 17 in K8s Dockerfile by default and remove `Dockerfile.java17`: Resolved
• Upgrade the minimum Minikube version to 1.28.0: Resolved
• Upgrade `kubernetes-client` to 6.4.0: Resolved
• Upgrade `kubernetes-client` to 6.3.1: Resolved
• Bump Kubernetes Client Version to 6.1.1: Resolved
• Support driver service labels: Resolved