[SPARK-13040] JDBC using SPARK_CLASSPATH is deprecated but is the only way documented

• Type: Documentation
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 2.0.0
• Component/s: Documentation, Examples
• Assignee: Sebastián Ramírez
• Reporter: Sebastián Ramírez
• Created: 27/Jan/16 10:42
• Updated: 09/Feb/16 08:51
• Resolved: 09/Feb/16 08:49

Description

The documentation says that to use a JDBC driver it must be set with the environment variable SPARK_CLASSPATH, as in:

SPARK_CLASSPATH=postgresql-9.3-1102-jdbc41.jar bin/spark-shell

http://spark.apache.org/docs/latest/sql-programming-guide.html#jdbc-to-other-databases

But when run like that, the output says that using that environment variable is deprecated:

SPARK_CLASSPATH was detected (set to '/home/senseta/postgresql-9.4.1207.jre7.jar').
This is deprecated in Spark 1.0+.

Please instead use:
 - ./spark-submit with --driver-class-path to augment the driver classpath
 - spark.executor.extraClassPath to augment the executor classpath

16/01/27 13:36:57 WARN spark.SparkConf: Setting 'spark.executor.extraClassPath' to '/home/senseta/postgresql-9.4.1207.jre7.jar' as a work-around.
16/01/27 13:36:57 WARN spark.SparkConf: Setting 'spark.driver.extraClassPath' to '/home/senseta/postgresql-9.4.1207.jre7.jar' as a work-around.

It would be good to have an example with the current official syntax (I'm actually not sure of which would be the correct parameters), for Scala and Python (in case they differ).

Issue Links

• links to: [Github] Pull Request #10948 (tiangolo)