[SPARK-24429] Add support for spark.driver.extraJavaOptions in cluster mode for Spark on K8s

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.4.0
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Stavros Kontopoulos
• Created: 30/May/18 13:59
• Updated: 17/May/20 18:24
• Resolved: 11/Oct/18 14:45

Description

 

Right now in cluster mode only extraJavaOptions targeting the executor are set. 

According to the implementation and the docs:

"In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file."

A typical driver launch in cluster mode will eventually use client mode to run the Spark-submit and looks like:

"/usr/lib/jvm/java-1.8-openjdk/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.SparkSubmit --deploy-mode client --conf spark.driver.bindAddress=9.0.7.116 --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi spark-internal 10000"

Also at the entrypoint.sh file there is no management of the driver's java opts. 

We propose to set an env var to pass the extra java opts to the driver (like in the case of the executor), rename the env vars in the container as the one for the executor is a bit misleading, and use --driver-java-options to pass the required options.