Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-49524

Improve K8s support

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 4.0.0
    • 4.0.0
    • Build, Kubernetes
    • None

    Attachments

      Issue Links

        is related to

        Umbrella - An overarching type made of sub-tasks SPARK-44111 Prepare Apache Spark 4.0.0

        • Critical - Crashes, loss of data, severe memory leak.
        • Open
        relates to

        Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-45869 Revisit and Improve Spark Standalone Cluster

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        New Feature - A new feature of the product, which has yet to be developed. SPARK-45923 Spark Kubernetes Operator

        • Major - Major loss of function.
        • Resolved
        1.
        		 Drop K8s v1.25 and lower version support Sub-task Resolved Dongjoon Hyun
        2.
        		 Drop K8s v1.26 Support Sub-task Resolved Dongjoon Hyun
        3.
        		 Drop K8s v1.27 Support Sub-task Resolved Dongjoon Hyun
        4.
        		 Drop K8s v1.28 Support Sub-task Resolved Dongjoon Hyun
        5.
        		 Add a symbolic link file `spark-examples.jar` in K8s Docker images Sub-task Resolved Dongjoon Hyun
        6.
        		 Support K8s volume `mount.subPathExpr` and `hostPath` volume `type` Sub-task Resolved Enrico Minack
        7.
        		 Support user-defined annotations for OnDemand PVCs Sub-task Resolved Prathit Malik
        8.
        		 Support AWS_ENDPOINT_URL env variable Sub-task Resolved Dongjoon Hyun
        9.
        		 Support APP_ID and EXECUTOR_ID placeholder in labels Sub-task Resolved Xi Chen
        10.
        		 Add `SPARK_EXECUTOR_ATTRIBUTE_(APP|EXECUTOR)_ID` if `CUSTOM_EXECUTOR_LOG_URL` is defined Sub-task Resolved Dongjoon Hyun
        11.
        		 Enable `spark.ui.prometheus.enabled` by default Sub-task Resolved Dongjoon Hyun
        12.
        		 Enable `spark.kubernetes.executor.checkAllContainers` by default Sub-task Resolved Dongjoon Hyun
        13.
        		 Skip deleting pod from k8s if the pod does not exists Sub-task Resolved leesf
        14.
        		 Use JDK for Spark 3.5+ Docker image Sub-task Resolved Dongjoon Hyun
        15.
        		 Use Java 21 instead of 21-jre in K8s Dockerfile Sub-task Resolved Dongjoon Hyun
        16.
        		 Publish Java 21 Docker image for preview1 Sub-task Resolved Dongjoon Hyun
        17.
        		 Make `spark-docker` repository's Dockerfiles up-to-date Sub-task Resolved Dongjoon Hyun
        18.
        		 Add `OpenContainers` Annotations in docker images Sub-task Resolved Dongjoon Hyun
        19.
        		 Fix to use the exit code of executor container always Sub-task Resolved Oleksiy Dyagilev
        20.
        		 Fix `RELEASE` file to have the correct information in Docker images Sub-task Resolved Dongjoon Hyun
        21.
        		 Fix `spark.ui.custom.executor.log.url` docs by adding K8s Sub-task Resolved Dongjoon Hyun
        22.
        		 Fix `ENV` key value format in K8s Dockerfiles Sub-task Resolved Dongjoon Hyun
        23.
        		 Fix `docker-image-tool.sh` to be up-to-date Sub-task Resolved Dongjoon Hyun
        24.
        		 Fix `spark.kubernetes.allocation.batch.delay` to prevent small values less than 100 Sub-task Resolved Dongjoon Hyun
        25.
        		 Fix `InvalidDefaultArgInFrom` in Python/R binding Dockerfiles Sub-task Resolved Dongjoon Hyun
        26.
        		 Remove `connector/docker` in favor of `Apache Spark Operator` Sub-task Resolved Dongjoon Hyun
        27.
        		 Upgrade kubernetes-client to 6.9.0 for K8s 1.28 Sub-task Resolved Dongjoon Hyun
        28.
        		 Upgrade `kubernetes-client` to 6.9.1 Sub-task Resolved Dongjoon Hyun
        29.
        		 Upgrade kubernetes-client to 6.10.0 for K8s v1.29.0 Sub-task Resolved Bjørn Jørgensen
        30.
        		 Upgrade kubernetes-client to 6.11.0 Sub-task Resolved Bjørn Jørgensen
        31.
        		 Upgrade `kubernetes-client` to 6.12.0 Sub-task Resolved Dongjoon Hyun
        32.
        		 Upgrade kubernetes-client to 6.13.0 for K8s v1.30.0 Sub-task Resolved Bjørn Jørgensen
        33.
        		 Upgrade `kubernetes-client` to 6.13.1 Sub-task Resolved BingKun Pan
        34.
        		 Upgrade `kubernetes-client` to 6.13.2 Sub-task Resolved BingKun Pan
        35.
        		 Upgrade kubernetes-client to 6.12.1 Sub-task Resolved Bjørn Jørgensen
        36.
        		 Upgrade `kubernetes-client` to 6.13.3 Sub-task Resolved Dongjoon Hyun
        37.
        		 Update kubernet-client to 6.13.4 Sub-task Resolved Bjørn Jørgensen
        38.
        		 Update `YuniKorn` docs with v1.4 Sub-task Resolved Dongjoon Hyun
        39.
        		 Update `YuniKorn` docs with v1.5.1 Sub-task Resolved Dongjoon Hyun
        40.
        		 Update `YuniKorn` docs with v1.5 Sub-task Resolved Dongjoon Hyun
        41.
        		 Update `YuniKorn` docs with v1.5.2 Sub-task Resolved Dongjoon Hyun
        42.
        		 Upgrade `Volcano` to 1.8.0 Sub-task Resolved Dongjoon Hyun
        43.
        		 Upgrade `Volcano` to 1.8.1 Sub-task Resolved Dongjoon Hyun
        44.
        		 Upgrade `Volcano` to 1.8.2 Sub-task Resolved Dongjoon Hyun
        45.
        		 Upgrade `Volcano` to 1.9.0 Sub-task Resolved Dongjoon Hyun
        46.
        		 `build_and_test.yml` should use `Volcano` 1.7.0 for `branch-3.4/3.5` Sub-task Resolved Dongjoon Hyun
        47.
        		 Use `built-in` storage classes in PVTestsSuite Sub-task Resolved Dongjoon Hyun
        48.
        		 Create and use a K8s test tag for `PersistentVolume` Sub-task Resolved Dongjoon Hyun
        49.
        		 Use the latest minikube in K8s IT Sub-task Resolved Dongjoon Hyun
        50.
        		 Add `VolumeSuite` to K8s IT Sub-task Resolved Dongjoon Hyun
        51.
        		 Update docs to mention delegation token support on K8s mode Sub-task Resolved Cheng Pan
        52.
        		 Update `publish-java17.yaml` and `publish-java21.yaml` to use `preview2` by default Sub-task Resolved Dongjoon Hyun
        53.
        		 Lower `KubernetesClusterSchedulerBackend.onDisconnected` log level to debug Sub-task Resolved Dongjoon Hyun
        54.
        		 Fix `getReusablePVCs` to use `podCreationTimeout` instead of `podAllocationDelay` Sub-task Resolved Dongjoon Hyun
        55.
        		 Make ExecutorPodsAllocatorSuite independent from default allocation batch size Sub-task Resolved Dongjoon Hyun
        56.
        		 Remove `clock` parameter from `DriverServiceFeatureStep` Sub-task Resolved Dongjoon Hyun
        57.
        		 Detect and fail on invalid volume sizes (< 1KiB) in K8s Sub-task Resolved Dongjoon Hyun
        58.
        		 Add `spark.kubernetes.legacy.useReadWriteOnceAccessMode` for old K8s clusters Sub-task Resolved Dongjoon Hyun
        59.
        		 Improve SBT K8s IT to skip R image build step if not needed Sub-task Resolved Dongjoon Hyun
        60.
        		 Fix MountVolumesFeatureStep to use ReadWriteOncePod instead of ReadWriteOnce Sub-task Resolved Dongjoon Hyun
        61.
        		 Use INFO log in ExecutorPodsWatcher.onClose if SparkContext is stopped Sub-task Resolved Dongjoon Hyun
        62.
        		 Document shuffle data recovery from the remounted K8s PVCs Sub-task Resolved Dongjoon Hyun
        63.
        		 Validate checksum of remounted PVC's shuffle data before recovery Sub-task Resolved Dongjoon Hyun
        64.
        		 K8s default service token file should not be materialized into token Sub-task Resolved Dongjoon Hyun
        65.
        		 Add ShuffleChecksumUtils.compareChecksums by reusing ShuffleChecksumTestHelp.compareChecksums Sub-task Resolved Dongjoon Hyun
        66.
        		 Promote SparkKubernetesClientFactory to DeveloperApi Sub-task Resolved Dongjoon Hyun
        67.
        		 VolcanoFeatureStep should not create `DefaultVolcanoClient` if not needed Sub-task Resolved Dongjoon Hyun
        68.
        		 ExecutorFailureTracker sometimes miss failed executors on k8s Sub-task Resolved Attila Zsolt Piros

        Activity

          People

            dongjoon Dongjoon Hyun
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: