Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33005

Kubernetes GA Preparation

    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.1
    • Component/s: Kubernetes
    • Labels:
    • Target Version/s:

      Attachments

        Issue Links

        1.
        Remove the words "experimental" in the k8s document Sub-task Resolved Takeshi Yamamuro
        2.
        Support --proxy-user for Spark on K8s Sub-task Resolved Pedro Rossi
        3.
        Allow using some alternatives for service accounts Sub-task Resolved Udbhav Agrawal
        4.
        Allow setting serviceAccountName for executor pods Sub-task Resolved Artsiom Yudovin
        5.
        ExecutorPodsLifecycleManager has sub-optimal behavior with dynamic allocation Sub-task Resolved Marcelo Masiero Vanzin
        6.
        ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars Sub-task Resolved Kousuke Saruta
        7.
        Driver cores in kubernetes are coupled with container resources, not spark.driver.cores Sub-task Resolved Onur Satici
        8.
        Fix flaky "Test basic decommissioning" test Sub-task Resolved Holden Karau
        9.
        Improve Spark Decommissioning K8s test relability Sub-task Resolved Holden Karau
        10.
        When processing new K8s state snapshots Spark treats Terminating nodes as terminated. Sub-task Resolved Holden Karau
        11.
        Decommissioning Integration Test checks for old string Sub-task Resolved Holden Karau
        12.
        Handle shuffle blocks being removed during decommissioning Sub-task Resolved Holden Karau
        13.
        Expose the ability for user to cleanup shuffle files Sub-task Resolved Holden Karau
        14.
        Support for Kubernetes NFS volume mounts Sub-task Resolved Seongjin Cho
        15.
        Support appId/execId placeholder in K8s SPARK_EXECUTOR_DIRS Sub-task Resolved Dongjoon Hyun
        16.
        Support execId placeholder in executor PVC conf Sub-task Resolved Dongjoon Hyun
        17.
        Support dynamic PVC creation/deletion for K8s executors Sub-task Resolved Dongjoon Hyun
        18.
        Support dynamic PVC creation/deletion for K8s driver Sub-task Resolved Dongjoon Hyun
        19.
        Remove Python2 test from K8s IT Sub-task Resolved Dongjoon Hyun
        20.
        Fix mismatched versions in test artifacts Sub-task Resolved Dongjoon Hyun
        21.
        Add dynamic PVC usage example into K8s doc Sub-task Resolved Dongjoon Hyun
        22.
        Fix typo of parameter in KubernetesTestComponents Sub-task Resolved merrily01
        23.
        Remove python2 installation in K8s python image Sub-task Resolved William Hyun
        24.
        Use unique ConfigMap name for executor pod template Sub-task Resolved Stijn De Haes
        25.
        Improve error message for insufficient K8s volume confs Sub-task Resolved German Schiavon Matteo
        26.
        Upgrade fabric8 to 4.10.3 Sub-task Resolved Jonathan Lafleche
        27.
        Fix & re-enable failing R K8s tests Sub-task Resolved Dongjoon Hyun
        28.
        Use LinkedHashMap instead of Map for newlyCreatedExecutors Sub-task Resolved Dongjoon Hyun
        29.
        Respect executor idle timeout conf in ExecutorPodsAllocator Sub-task Resolved Dongjoon Hyun
        30.
        Make spark.kubernetes.pyspark.pythonVersion allow `Python 3` only Sub-task Resolved Dongjoon Hyun
        31.
        Detect duplicated mountPath and fail at Spark side Sub-task Resolved Dongjoon Hyun
        32.
        Use 11-jre-slim as default in K8s Dockerfile Sub-task Resolved Dongjoon Hyun
        33.
        Use default Hadoop profile by removing explicit `-Phadoop-2.7` from K8s IT Jenkins job Sub-task Resolved Dongjoon Hyun
        34.
        Upgrade kubernetes-client to 4.11.1 Sub-task Resolved Dongjoon Hyun
        35.
        Use R 3.6.3 in K8s R image Sub-task Resolved Dongjoon Hyun
        36.
        Spark client terminates while driver is still running Sub-task Resolved Stijn De Haes
        37.
        Spark executors on K8S do not request extra memory for off-heap allocations Sub-task Closed Unassigned
        38.
        Kubernetes Configuration to set Service Account to Executors Sub-task Closed Unassigned
        39.
        getMigrationBlocks should not fail at missing files Sub-task Resolved Dongjoon Hyun
        40.
        Support ordered shuffle block migration Sub-task Resolved Dongjoon Hyun
        41.
        Fix BlockManagerDecommissioner to return the correct migration status Sub-task Resolved Dongjoon Hyun
        42.
        Update misleading log messages for removed shuffle blocks Sub-task Resolved Dongjoon Hyun
        43.
        Add StorageLevel.DISK_ONLY_3 Sub-task Resolved Dongjoon Hyun
        44.
        Upgrade kubernetes-client to 4.12.0 Sub-task Resolved rameshkrishnan muthusamy
        45.
        Support local dependency uploading from --py-files Sub-task Resolved Stavros Kontopoulos
        46.
        Support Fallback Storage during Worker decommission Sub-task Resolved Dongjoon Hyun
        47.
        Make spark.archives working in Kubernates Sub-task Resolved Hyukjin Kwon
        48.
        Allow k8s integration tests to assert both driver and executor logs for expected log(s) Sub-task Resolved Prashant Sharma
        49.
        Support PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables Sub-task Resolved Hyukjin Kwon
        50.
        Decommissioning Race Condition during Pod Snapshot Sub-task Resolved Holden Karau
        51.
        Update dynamic allocation related documents Sub-task Resolved Dongjoon Hyun
        52.
        Exclude fallback block manager from executorList Sub-task Resolved Dongjoon Hyun
        53.
        Fix fallback storage offset and improve compression codec test coverage Sub-task Resolved Dongjoon Hyun
        54.
        NPE in ExecutorPodsSnapshot Sub-task Resolved Holden Karau
        55.
        Race condition in Spark k8s Pod lifecycle manager that leads to shutdowns Sub-task Resolved Attila Zsolt Piros
        56.
        Spark may report PodRunning if there is a sidecar that has not exited Sub-task Resolved Holden Karau
        57.
        Update kubernetes/integration-tests/README.md to follow the default Hadoop profile updated Sub-task Resolved Kousuke Saruta
        58.
        Kubernetes integration tests doesn't work with Minikube 1.9+ Sub-task Resolved Kousuke Saruta
        59.
        `gpg: keyserver receive failed: No name` during K8s IT Sub-task Resolved Holden Karau
        60.
        Fix flaky test "Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties." Sub-task Resolved Prashant Sharma
        61.
        Support k8s cluster manager with stage level scheduling Sub-task Resolved Thomas Graves
        62.
        Keep pending pods in account while scheduling new pods Sub-task Resolved Holden Karau
        63.
        Long runtime on many environment variables Sub-task Resolved Norbert Schultz
        64.
        Fix flaky test in DecommissionWorkerSuite Sub-task Resolved XiDuo You
        65.
        Promote spark.kubernetes.executor.podNamePrefix to the public conf Sub-task Resolved Dongjoon Hyun
        66.
        KubernetesClusterSchedulerBackend.stop should clean up K8s resources Sub-task Resolved Dongjoon Hyun
        67.
        ExecutorPodsAllocator fails to identify some excess requests during downscaling Sub-task Resolved Attila Zsolt Piros
        68.
        K8s integration test should use the runtime Hadoop Version Sub-task Resolved Dongjoon Hyun
        69.
        Ignore RegisterExecutor when SparkContext is stopped Sub-task Resolved Dongjoon Hyun

          Activity

            People

            • Assignee:
              dongjoon Dongjoon Hyun
              Reporter:
              dongjoon Dongjoon Hyun
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: