[SPARK-1753] PySpark on YARN does not work on assembly jar built on Red Hat based OS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.0
• Component/s: PySpark, Spark Core
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 08/May/14 00:36
• Updated: 03/Mar/15 14:40
• Resolved: 13/May/14 02:44

Description

If the jar is built on a Red Hat based OS, the additional python files included in the jar cannot be accessed. This means PySpark doesn't work on YARN because in this mode it relies on the python files within this jar.

I have confirmed that my Java, Scala, and maven versions are all exactly the same on my CentOS environment and on my local OSX environment, and the former does not work. Thomas Graves also struggled with the same problem.

Until a fix is found, we should at the very least document this peculiarity.

Issue Links

• is related to: SPARK-1911 Warn users if their assembly jars are not built with Java 6