Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12559

Cluster mode doesn't work with --packages

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Submit
    • Labels:
      None

      Description

      From the mailing list:

      Another problem I ran into that you also might is that --packages doesn't
      work with --deploy-mode cluster. It downloads the packages to a temporary
      location on the node running spark-submit, then passes those paths to the
      node that is running the Driver, but since that isn't the same machine, it
      can't find anything and fails. The driver process should be the one
      doing the downloading, but it isn't. I ended up having to create a fat JAR
      with all of the dependencies to get around that one.

      The problem is that we currently don't upload jars to the cluster. It seems to fix this we either (1) do upload jars, or (2) just run the packages code on the driver side. I slightly prefer (2) because it's simpler.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                skonto Stavros Kontopoulos
                Reporter:
                andrewor14 Andrew Or
              • Votes:
                2 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: