[SPARK-21798] No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: Spark Core
• Assignee: Parth Gandhi
• Reporter: Sanket Reddy
• Created: 21/Aug/17 14:35
• Updated: 28/Aug/17 13:52
• Resolved: 28/Aug/17 13:52

Description

History Server Launch uses SparkClassCommandBuilder for launching the server. It is observed that SPARK_CLASSPATH has been removed and deprecated. For spark-submit this takes a different route and spark.driver.extraClasspath takes care of specifying additional jars in the classpath that were previously specified in the SPARK_CLASSPATH. Right now the only way specify the additional jars for launching daemons such as history server is using SPARK_DIST_CLASSPATH (https://spark.apache.org/docs/latest/hadoop-provided.html) but this I presume is a distribution classpath. It would be nice to have a similar config like spark.driver.extraClasspath for launching daemons similar to history server.

Issue Links

• links to: [Github] Pull Request #19047 (pgandhi999)