[SPARK-6047] pyspark - class loading on driver failing with --jars and --packages

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.3.0
• Component/s: PySpark, Spark Submit
• Assignee: Unassigned
• Reporter: Burak Yavuz
• Created: 27/Feb/15 00:08
• Updated: 13/Mar/16 21:44
• Resolved: 13/Mar/16 21:44

Description

Because py4j uses the system ClassLoader instead of the contextClassLoader of the thread, the dynamically added jars in Spark Submit can't be loaded in the driver.

This causes `Py4JError: Trying to call a package` errors.

Usually `-packages` are downloaded from some remote repo before runtime, adding them explicitly to `driver-class-path` is not an option, like we can do with `jars`. One solution is to move the fetching of `-packages` to the SparkSubmitDriverBootstrapper, and add it to the driver class-path there.

A more complete solution can be achieved through SPARK-4924.

Issue Links

• duplicates: SPARK-5185 pyspark --jars does not add classes to driver class path
• incorporates: SPARK-5185 pyspark --jars does not add classes to driver class path
• is related to: SPARK-4924 Factor out code to launch Spark applications into a separate library
• links to: [Github] Pull Request #4754 (brkyvz)