[SPARK-3734] DriverRunner should not read SPARK_HOME from submitter's environment

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0, 1.2.0
• Fix Version/s: 1.1.1, 1.2.0
• Component/s: Deploy
• Labels: pull-request-available
• Target Version/s: 1.1.1, 1.2.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 29/Sep/14 23:19
• Updated: 28/Jan/24 16:13
• Resolved: 30/Sep/14 06:37

Description

If you use spark-submit in cluster mode to submit a job to a Spark Standalone cluster, and the JAVA_HOME environment variable is set on the submitting machine, then DriverRunner will attempt to use the submitter's JAVA_HOME to launch the driver process (instead of the worker's JAVA_HOME), which can cause the job to fail unless the submitter and worker have Java installed in the same location.

This has a pretty simple fix: read JAVA_HOME from sys.env instead of command.environment; PR pending shortly.

Issue Links

• links to: [Github] Pull Request #2586 (JoshRosen)
• links to: GitHub Pull Request #2586