[ZEPPELIN-1741] JAR's specified with spark.jars in spark-defaults.conf does not affect %pyspark interpreter

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.6.2
• Component/s: Interpreters

Environment

Zeppelin 0.6.2
Spark 1.6.2

• Assignee: Unassigned
• Reporter: JJ Gangi
• Created: 30/Nov/16 19:25
• Updated: 14/Jan/17 06:48

Description

If you specify JAR's with spark.jars in your spark-defaults.conf, the %pyspark interpreter will not load these JARs.

Currently, a note in the Spark interpreter documentation says, "Note that adding jar to pyspark is only availabe via %dep interpreter at the moment."

This is undesirable for two reasons:

1) %dep is deprecated
2) Sysadmins configure spark-defaults.conf, and expect these settings to be honored however Spark is executed.

As a Zeppelin user, I expect that if I configure JAR's in spark-defaults.conf, these JAR's will be available when the %pyspark interpreter is run.

Issue Links

• relates to: ZEPPELIN-1883 Can't import packages requested by SPARK_SUBMIT_OPTION in pyspark