[SPARK-23768] Proxy configuration for extraJavaOptions in defaults conf

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.2.1
• Component/s: Spark Shell
• Labels: bulk-closed

Environment

default conf setting:

spark.executor.extraJavaOptions    -Dhttp.proxyHost=IP -Dhttp.proxyPort=8080 -Dhttps.proxyHost=IP -Dhttps.proxyPort=8080

spark.jars.packages                datastax:spark-cassandra-connector:2.0.0-M2-s_2.11

 

• Assignee: Unassigned
• Reporter: Praveen
• Created: 22/Mar/18 11:35
• Updated: 21/May/19 04:12
• Resolved: 21/May/19 04:12

Description

When trying to launch spark shell or pyspark for the first time with cassandra connector configured as package, the proxy setting in default conf is not being used and the download fails (if behind proxy).

If the proxy is configured directly in the command line, the download happens and spark shell starts correctly. 

Seems the proxy configuration in default conf is not being used for package download.