[SPARK-45923] Spark Kubernetes Operator

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: kubernetes-operator-0.1.0
• Fix Version/s: 4.0.0
• Component/s: Kubernetes
• Labels: SPIP pull-request-available releasenotes
• Assignee: Zhou Jiang
• Reporter: Zhou Jiang
• Shepherd: L. C. Hsieh
• Created: 14/Nov/23 17:38
• Updated: 20/Sep/24 11:56
• Resolved: 28/Aug/24 17:21

Description

We would like to develop a Java-based Kubernetes operator for Apache Spark. Following the operator pattern (https://kubernetes.io/docs/concepts/extend-kubernetes/operator/), Spark users may manage applications and related components seamlessly using native tools like kubectl. The primary goal is to simplify the Spark user experience on Kubernetes, minimizing the learning curve and operational complexities and therefore enable users to focus on the Spark application development.

Ideally, it would reside in a separate repository (like Spark docker or Spark connect golang) and be loosely connected to the Spark release cycle while supporting multiple Spark versions.

SPIP doc: https://docs.google.com/document/d/1f5mm9VpSKeWC72Y9IiKN2jbBn32rHxjWKUfLRaGEcLE

Dev email discussion : https://lists.apache.org/thread/wdy7jfhf7m8jy74p6s0npjfd15ym5rxz

Issue Links

• is related to: SPARK-44111 Prepare Apache Spark 4.0.0
• is related to: SPARK-49524 Improve K8s support
• relates to: SPARK-45869 Revisit and Improve Spark Standalone Cluster

Sub-Tasks

• Add License to Spark Operator: Resolved
• Setup gradle as build tool for operator repository: Resolved
• Setup Static Analysis for Operator: Resolved
• Add Operator CI Task for Java Build and Test: Resolved
• Add Java API Module for Spark Operator: Resolved
• Use API Group `spark.apache.org`: Resolved
• Update `build.gradle` to fix deprecation warnings: Resolved
• Promote `KubernetesVolumeUtils` to `DeveloperApi`: Resolved
• Promote `KubernetesClientUtils` to `DeveloperApi`: Resolved
• Promote `o.a.s.d.k8s.Constants` to `DeveloperApi`: Resolved
• Promote `*MainAppResource` and `NonJVMResource` to `DeveloperApi`: Resolved
• Promote ` KubernetesDriverBuilder` to `DeveloperApi`: Resolved
• Promote ` KubernetesDriverSpec` to `DeveloperApi`: Resolved
• Promote ` KubernetesDriverConf` to `DeveloperApi`: Resolved
• Promote `PrometheusServlet` to `DeveloperApi`: Resolved
• Add Spark application submission worker for operator: Resolved
• Enable autolink to SPARK jira issue: Resolved
• Use the official Apache Spark 4.0.0-preview1: Resolved
• Upgrade checkstyle and spotbugs version: Resolved
• Define Config Loading Framework for Spark Operator Controller: Resolved
• Use `BasePluginExtension` in `spark-operator/build.gradle`: Resolved
• Upgrade Gradle to 8.9: Resolved
• Upgrade Gradle to 8.10: Resolved
• Avoid unnecessary task configuration in `spark-operator-api`: Resolved
• Fix `spark-operator` module to define test framework explicitly: Resolved
• Add `reconciler` to `spark-operator` module: Resolved
• Ban `org.apache.commons.collections` in favor of `collections4`: Resolved
• Add Controller Metrics System and Utils: Resolved
• Use `try-with-resources` to test `KubernetesClientFactory.buildKubernetesClient`: Resolved
• Avoid `raw` type usage: Resolved
• Exclude `JUnitAssertionsShouldIncludeMessage/JUnitTestContainsTooManyAsserts` PMD rules and simplify test code: Resolved
• Add docker image build for operator: Resolved
• Add SparkOperator class and tests: Resolved
• Fix `ENTRYPOINT` to point `/opt/spark-operator/operator/docker-entrypoint.sh`: Resolved
• Verify built images in `build-image` CI job via `docker run` test: Resolved
• Minimize docker image by removing redundant `chown` commands: Resolved
• Remove `commons-lang3` dependency from `spark-operator-api`: Resolved
• Reduce `spark-operator` fat jar size by excluding dependencies: Resolved
• Use the latest `setup-java` v4 with `cache` feature: Resolved
• Use the latest PMD 6.x rules instead of the deprecated ones: Resolved
• Increase `Gradle` JVM memory to `4g` like Spark repo: Resolved
• Enforce ImmutableField and UselessParentheses rules: Resolved
• Enforce SignatureDeclareThrowsException and AvoidThrowingRawExceptionTypes rules: Resolved
• Enforce ConfusingTernary and PrematureDeclaration rules: Resolved
• Enforce `FieldDeclarationsShouldBeAtStartOfClass`, `LinguisticNaming` and `ClassWithOnlyPrivateConstructorsShouldBeFinal` rules: Resolved
• Fix RestartPolicyTest to cover `SchedulingFailure`: Resolved
• Enforce UseUtilityClass rule: Resolved
• Add `OpenContainers` Annotations to docker image: Resolved
• Update `README.md` with `build/test/CI` info: Resolved
• Speed up docker image building by excluding `check` instead of `test`: Resolved
• Add `k8s-integration-tests` GitHub Action CI job: Resolved
• Add Helm Chart: Resolved
• Install and test Helm chart in K8s integration test CI: Resolved
• Upgrade `kubernetes-client` to 6.13.3 and `commons-lang3` to 3.16.0: Resolved
• Add `pi.yaml` example and update README.md: Resolved
• Add `pi-with-one-pod.yaml` example: Resolved
• Add `pi-on-yunikorn.yaml` example: Resolved
• Add `sql.yaml` example: Resolved
• Improve `gradlew` to support both `curl` and `wget`: Resolved
• Refactor `RestartPolicyTest` to test per case: Resolved
• Revise `reconcilesteps` package and `SparkAppReconciler`: Resolved
• Revise `observers` package: Resolved
• Fix `docker-entrypoint.sh` to quote the environment variables: Resolved
• Fix `javadoc` generation and add `lint` test pipeline to prevent: Resolved
• Add `_MESSAGE` postfix to `DRIVER_(READY|RUNNING)`: Resolved
• Add `NOTICE`, `NOTICE-binary`, `LICENSE-binary` files and update `*.gradle` files: Resolved
• Add `deploy.gradle` to support publish-related tasks: Resolved
• Add `buildDockerImage` Gradle Task: Resolved
• Add `-SNAPSHOT` postfix to `Spark Operator` version: Resolved
• Revise `Spark Operator` docker image: Resolved
• Use `HTTP_*` constant variables instead of magic numbers: Resolved
• Generalize `relocateGeneratedCRD` Gradle Task to handle `*.spark.apache.org-v1.yml`: Resolved
• Generalize `printer-columns.sh` to handle `*.spark.apache.org-v1.yml` files: Resolved
• Add `SparkCluster` to `spark-operator-api` module and examples: Resolved
• Add `SparkCluster` to `spark-submission-worker` module: Resolved
• Add `SparkCluster` to `spark-operator` module: Resolved
• Propagate Spark configurations to SparkCluster: Resolved
• Use `setup-minikube` GitHub Action: Resolved
• Add K8s service for Workers to SparkClusterResourceSpec: Resolved
• Revise `InstanceConfig` to `ExecutorInstanceConfig` class: Resolved
• Add e2e test in operator workflow: Resolved
• Document `SparkCluster` and add `submit-pi-to-prod.sh` example: Resolved
• Document nightly versions of operator image and Helm Chart: Resolved
• Add `publish_snapshot_dockerhub.yml` Daily GitHub Action job: Resolved
• Support `schedulerName` for SparkCluster: Resolved
• Add `publish_snapshot_chart.yml` GitHub Action job: Resolved
• Use `rsync` to upload to `nightlies`: Resolved
• Fix `Dockerfile` by removing unused `ARG` from builder and moving default value: Resolved
• Simplify snapshot HelmChart to use `apache/spark-kubernetes-operator:main-snapshot` by default: Resolved
• Introduce ClusterToleration and WorkerInstanceConfig: Resolved
• E2E Tests Catch Step SparkApplication Kind Not found: Resolved
• Enable Pull Request Labeler: Resolved
• Support user provided spec for SparkCluster: Resolved
• Add `cluster-with-template.yaml`: Resolved
• E2E Test Components: State Transition: Resolved
• Remove `SPARK_NO_DAEMONIZE` in favor of live log UIs: Resolved
• Support `master|worker` container templates: Resolved
• Add `8081` port to Worker resource spec: Resolved
• Add `Clean Up` section to `README.md`: Resolved
• Use Gradle Version Catalog: Resolved
• Add `log4j2` default setting to `values.yaml`: Resolved
• Add `pi-scala.yaml` and `pyspark-pi.yaml`: Resolved
• Adjust `ERROR`-level log messages: Resolved
• Hot Properties Reload E2E Test: Resolved
• Fix K8s version in GitHub Action CI: Resolved
• Upgrade the minimum K8s version to v1.28: Resolved
• Generate Spark Operator Config Property Doc: Resolved
• E2E test template includes invalid spec field: Resolved
• Upgrade Gradle to 8.10.1: Resolved
• Refactor prefix `appResources` to `workloadResources`: Resolved
• Spark Cluster Happy Path State Transition Test: Resolved
• Update `e2e/python/chainsaw-test.yaml` to use non-R image: Resolved
• Two instances of Spark Operator running at the same time: Resolved
• E2E Workflow Refactor: Resolved
• Use `apache/spark` images instead of `spark`: Resolved
• Use `spark-examples.jar` instead of `spark-examples_2.13-4.0.0-preview1.jar`: Resolved
• Add `Java 21`-based `SparkPi` example: Resolved
• Add `Java 21`-based `SparkCluster` example: Resolved
• Add Java 21 in the e2e test: Resolved
• Upgrade Spark to `4.0.0-preview2`: Resolved
• Upgrade `README`, examples, tests to use `preview2`: Resolved
• Support HPA for `SparkCluster`: Resolved
• Support `HPA` template for `SparkCluster`: Resolved
• Remove (master|worker) prefix from field names of `(Master|Worker)Spec`: Resolved
• Update E2E tests to use Spark 3.5.3: Resolved
• Fix the error when enable the sparkApplicationSentinel: Resolved
• Provide empty `RuntimeVersions` object to `ClusterSpec.runtimeVersions` by default: Resolved
• Make `o.a.s.k8s.operator.utils.Utils` argument naming consistent: Resolved
• Add `spark-version` label to `Spark Cluster` resources: Resolved
• Add docs for operator: Resolved
• Update `cluster-with-template.yaml` example with pod annotation: Resolved
• Use `registry.k8s.io/pause:3.9` to avoid failure deterministically: Resolved
• Upgrade Gradle to 8.11: Resolved
• Update docs with YuniKorn 1.6.0 and first-time installation guide: Resolved
• Upgrade `kubernetes-client` to 6.13.4 and `log4j` to 2.24.2: Resolved