Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34870

Jars downloaded with the --packages argument are not added to the classpath for executors.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Spark Submit
    • Labels:
      None
    • Environment:

      Spark worker running inside a Kubernetes pod with a Bitnami Spark image, and the driver running inside of a Jupyter Spark Kubernetes pod.

      Description

      When Spark is run in local mode, it works as expected. However, when Spark is run in client mode, it copies the jars to the executor ($SPARK_HOME/work/<app id>/<executor id>), but never adds them to the classpath.

      It might be worth noting that `spark.jars` does add the jars to the classpath, but unlike `spark.jars.packages` it doesn't automatically download the jar's compile dependencies.

       

      ```

      spark = SparkSession.builder\
      .master(SPARK_MASTER)\
      .appName(APP_NAME)\
      ...
      .config("spark.jars.packages", DEPENDENCY_PACKAGES) \

      ...
      .getOrCreate()
      ```

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cjmaklin Cory Maklin
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: