[SPARK-35072] spark.jars.ivysettings should support local:// and hdfs:// schemes

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Shardul Mahadik
• Created: 14/Apr/21 15:08
• Updated: 14/Apr/21 15:14

Description

During reviews of SPARK-34472, there was a desire to support local:// and hdfs:// schemes and potentially other schemes with spark.jars.ivysettings. See https://github.com/apache/spark/pull/31591#discussion_r598850998 and https://github.com/apache/spark/pull/31591#issuecomment-817951152. Currently this fails with the following error:

./bin/spark-shell --conf spark.jars.packages="org.apache.commons:commons-lang:3.4" --conf spark.jars.ivySettings="local:///Use
rs/test/temp/ivysettings.xml"
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Ivy settings file local:/Users/test/temp/ivysettings.xml does not exist
	at scala.Predef$.require(Predef.scala:281)
	at org.apache.spark.deploy.SparkSubmitUtils$.loadIvySettings(SparkSubmit.scala:1288)
	at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:176)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:895)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1031)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1040)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
 

We should make sure that this also works with SparkContext#addJar when running in cluster mode. At this point, only YARN resource manager supports managing ivysettings file in cluster mode. This can change based on SPARK-35073.

Issue Links

• relates to: SPARK-34472 SparkContext.addJar with an ivy path fails in cluster mode with a custom ivySettings file