[SPARK-11085] Add support for HTTP proxy

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Component/s: Spark Shell, Spark Submit
• Assignee: Unassigned
• Reporter: Dustin Cote
• Created: 13/Oct/15 14:01
• Updated: 11/Dec/18 22:51
• Resolved: 11/Dec/18 22:51

Description

Add a way to update ivysettings.xml for the spark-shell and spark-submit to support proxy settings for clusters that need to access a remote repository through an http proxy. Typically this would be done like:
JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=proxy.host -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.host.secure -Dhttps.proxyPort=8080"

Directly in the ivysettings.xml would look like:
<ivysettings>
<setproxy proxyhost="proxy.host"
proxyport="8080"
nonproxyhosts="nonproxy.host"/>
</ivysettings>

Even better would be a way to customize the ivysettings.xml with command options.

Issue Links

• duplicates: SPARK-17568 Add spark-submit option for user to override ivy settings used to resolve packages/artifacts