[SPARK-29487] Ability to run Spark Kubernetes other than from /opt/spark

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.4.4
• Component/s: Kubernetes, Spark Core, Spark Submit
• Assignee: Unassigned
• Reporter: Benjamin Miao CAI
• Created: 16/Oct/19 06:21
• Updated: 17/May/20 18:24
• Resolved: 12/Nov/19 19:18

Description

On spark kubernetes Dockerfile, the spark binaries are copied to /opt/spark.

If we try to create our own Dockerfile without using /opt/spark then the image will not run.

After looking at the source code, it seem that in various places, the path is hard-coded to /opt/spark

Example :

Constants.scala :

// Spark app configs for containers
val SPARK_CONF_VOLUME = "spark-conf-volume"
val SPARK_CONF_DIR_INTERNAL = "/opt/spark/conf"

 

Is it possible to make this configurable so we can put spark elsewhere than /opt/.

 

 

Issue Links

• duplicates: SPARK-24655 [K8S] Custom Docker Image Expectations and Documentation