Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35072

spark.jars.ivysettings should support local:// and hdfs:// schemes

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.2.0
    • None
    • Spark Core
    • None

    Description

      During reviews of SPARK-34472, there was a desire to support local:// and hdfs:// schemes and potentially other schemes with spark.jars.ivysettings. See https://github.com/apache/spark/pull/31591#discussion_r598850998 and https://github.com/apache/spark/pull/31591#issuecomment-817951152. Currently this fails with the following error:

      ./bin/spark-shell --conf spark.jars.packages="org.apache.commons:commons-lang:3.4" --conf spark.jars.ivySettings="local:///Use
      rs/test/temp/ivysettings.xml"
      Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Ivy settings file local:/Users/test/temp/ivysettings.xml does not exist
      	at scala.Predef$.require(Predef.scala:281)
      	at org.apache.spark.deploy.SparkSubmitUtils$.loadIvySettings(SparkSubmit.scala:1288)
      	at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:176)
      	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
      	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:895)
      	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
      	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
      	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
      	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1031)
      	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1040)
      	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
       

      We should make sure that this also works with SparkContext#addJar when running in cluster mode. At this point, only YARN resource manager supports managing ivysettings file in cluster mode. This can change based on SPARK-35073.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            shardulm Shardul Mahadik

            Dates

              Created:
              Updated:

              Slack

                Issue deployment