Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1652 Fixes and improvements for spark-submit/configs
  3. SPARK-1870

Jars specified via --jars in spark-submit are not added to executor classpath for YARN

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 0.9.2, 1.0.0
    • YARN
    • None

    Description

      With `spark-submit`, jars specified via `--jars` are added to distributed cache in `yarn-cluster` mode. The executor should add cached jars to classpath. However,

      sc.parallelize(0 to 10, 10).map { i =>
        System.getProperty("java.class.path")
      }.collect().foreach(println)
      

      shows only system jars, `app.jar`, and `spark.jar` but not other jars in the distributed cache.

      The workaround is using assembly jar.

      Attachments

        Activity

          People

            mengxr Xiangrui Meng
            mengxr Xiangrui Meng
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: