Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24434

Support user-specified driver and executor pod templates

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • Kubernetes, Spark Core
    • None

    Description

      With more requests for customizing the driver and executor pods coming, the current approach of adding new Spark configuration options has some serious drawbacks: 1) it means more Kubernetes specific configuration options to maintain, and 2) it widens the gap between the declarative model used by Kubernetes and the configuration model used by Spark. We should start designing a solution that allows users to specify pod templates as central places for all customization needs for the driver and executor pods. 

      Attachments

        Issue Links

          duplicates

          New Feature - A new feature of the product, which has yet to be developed. SPARK-24435 Support user-supplied YAML that can be merged with k8s pod descriptions

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Sub-task - The sub-task of the issue SPARK-39026 Add k8s tolerations support for apache spark

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-24491 Support for requesting GPU resources on K8S

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-39023 Add Executor Pod inter-pod anti-affinity

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-39025 Add k8s Scheduling, Preemption and Eviction feature to apache spark

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-25220 [K8S] Split out node selector config between driver and executors.

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-27334 Support specify scheduler name for executor pods when submit

          • Major - Major loss of function.
          • Closed
          is related to

          Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' SPARK-24091 Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files

          • Major - Major loss of function.
          • Resolved
          supercedes

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-25053 Allow additional port forwarding on Spark on K8S as needed

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed
          links to

          Pull request #22146 [Github] Pull Request #22146 (onursatici)

          Activity

            People

              onursatici Onur Satici
              liyinan926 Yinan Li
              Votes:
              2 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: