Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24091

Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files

    XMLWordPrintableJSON

Details

    • Brainstorming
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.4.0
    • None
    • Kubernetes, Spark Core

    Description

      The recent PR https://github.com/apache/spark/pull/20669 for removing the init-container introduced a internally used ConfigMap carrying Spark configuration properties in a file for the driver. This ConfigMap gets mounted under $SPARK_HOME/conf and the environment variable SPARK_CONF_DIR is set to point to the mount path. This pretty much prevents users from mounting their own ConfigMaps that carry custom Spark configuration files, e.g., log4j.properties and spark-env.sh and leaves users with only the option of building custom images. IMO, it is very useful to support mounting user-specified ConfigMaps for custom Spark configuration files. This worths further discussions.

      Attachments

        Issue Links

          Is contained by

          Umbrella - An overarching type made of sub-tasks SPARK-24090 Kubernetes Backend Hotlist for Spark 2.4

          • Major - Major loss of function.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-24434 Support user-specified driver and executor pod templates

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              liyinan926 Yinan Li
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: