[SPARK-38396] Improve K8s Integration Tests

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: Kubernetes, Tests
• Labels: releasenotes
• Target Version/s: 3.3.0
• Assignee: William Hyun
• Reporter: Dongjoon Hyun
• Created: 02/Mar/22 19:40
• Updated: 09/Mar/22 19:48
• Resolved: 02/Mar/22 20:13

Description

This JIRA aims to improve K8s integration tests for the following recent activity.

• Support Java 17 / Native Apple Silicon
• Cloud K8s testing (including Graviton instances and K8s 1.22 GA on EKS on March 15th)
• New K8s model (K8s clients)
• New custom schedulers and statefulset

Issue Links

• relates to: SPARK-33772 Build and Run Spark on Java 17
• relates to: SPARK-35781 Support Spark on Apple Silicon on macOS natively on Java 17
• relates to: SPARK-36057 SPIP: Support Customized Kubernetes Schedulers

Sub-Tasks

• Support docker-desktop K8S integration test in SBT: Resolved
• Add IntegrationTestBackend.describePods to support all K8s test backends: Resolved
• Support K8s namespace parameter in SBT K8s IT: Resolved
• Support K8s imageTag parameter in SBT K8s IT: Resolved
• Add SBT commands to K8s IT readme: Resolved
• Support K8s image building with Java 17: Resolved
• Use relativePath for K8s remote file test in BasicTestsSuite: Resolved
• Support cloud-backend in K8s IT with SBT: Resolved
• Add `priorityClassName` integration test case: Resolved
• Upgrade the minimum Minikube version to 1.18.0: Resolved
• Fix driver environment verification in BasicDriverFeatureStepSuite: Resolved
• Make the Java version installed on the container image used by the K8s integration tests with SBT configurable: Resolved
• Support K8s integration tests for Java 17: Resolved
• Add `spark-` prefix to namespaces and `-driver` suffix to drivers during IT: Resolved
• Close KubernetesClient in K8S integrations tests: Resolved
• Upgrade kubernetes-client to 5.12.1: Resolved
• Use docker-desktop instead of docker-for-desktop for Docker K8S IT deployMode and context name: Resolved
• Use `rbac.authorization.k8s.io/v1` instead of `v1beta1`: Resolved
• Support pyDockerfile and rDockerfile in SBT K8s IT: Resolved
• Word spell error - "labeled" spells as "labled": Resolved
• Use Java 17 in K8S integration tests when setting spark-tgz: Resolved