[SPARK-16781] java launched by PySpark as gateway may not be the same java used in the spark environment

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.2
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: PySpark
• Assignee: Sean R. Owen
• Reporter: Michael Berman
• Created: 28/Jul/16 18:34
• Updated: 24/Aug/16 19:05
• Resolved: 24/Aug/16 19:04

Description

When launching spark on a system with multiple javas installed, there are a few options for choosing which JRE to use, setting `JAVA_HOME` being the most straightforward.

However, when pyspark's internal py4j launches its JavaGateway, it always invokes `java` directly, without qualification. This means you get whatever java's first on your path, which is not necessarily the same one in spark's JAVA_HOME.

This could be seen as a py4j issue, but from their point of view, the fix is easy: make sure the java you want is first on your path. I can't figure out a way to make that reliably happen through the pyspark executor launch path, and it seems like something that would ideally happen automatically. If I set JAVA_HOME when launching spark, I would expect that to be the only java used throughout the stack.

Issue Links

• is duplicated by: SPARK-17220 Upgrade Py4J to 0.10.3
• links to: [Github] Pull Request #14748 (srowen)