[SPARK-12559] Cluster mode doesn't work with --packages

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 2.3.0
• Component/s: Spark Submit
• Assignee: Stavros Kontopoulos
• Reporter: Andrew Or
• Created: 29/Dec/15 18:50
• Updated: 09/May/18 20:41
• Resolved: 11/Aug/17 23:02

Description

From the mailing list:

Another problem I ran into that you also might is that --packages doesn't
work with --deploy-mode cluster. It downloads the packages to a temporary
location on the node running spark-submit, then passes those paths to the
node that is running the Driver, but since that isn't the same machine, it
can't find anything and fails. The driver process should be the one
doing the downloading, but it isn't. I ended up having to create a fat JAR
with all of the dependencies to get around that one.

The problem is that we currently don't upload jars to the cluster. It seems to fix this we either (1) do upload jars, or (2) just run the packages code on the driver side. I slightly prefer (2) because it's simpler.

Issue Links

• is cloned by: SPARK-21403 Cluster mode doesn't work with --packages [Mesos]
• is duplicated by: SPARK-6031 Refactor --packages to work inside the DriverBootstrapper so that the jars can be added to the driver classpath
• links to: [Github] Pull Request #18587 (skonto)
• links to: [Github] Pull Request #18630 (skonto)