[SPARK-34472] SparkContext.addJar with an ivy path fails in cluster mode with a custom ivySettings file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Spark Core
• Assignee: Shardul Mahadik
• Reporter: Shardul Mahadik
• Created: 19/Feb/21 10:00
• Updated: 20/Apr/21 18:37
• Resolved: 20/Apr/21 18:37

Description

SPARK-33084 introduced support for Ivy paths in sc.addJar or Spark SQL ADD JAR. If we use a custom ivySettings file using spark.jars.ivySettings, it is loaded at https://github.com/apache/spark/blob/b26e7b510bbaee63c4095ab47e75ff2a70e377d7/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L1280. However, this file is only accessible on the client machine. In cluster mode, this file is not available on the driver and so addJar fails.

spark-submit --master yarn --deploy-mode cluster --class IvyAddJarExample --conf spark.jars.ivySettings=/path/to/ivySettings.xml example.jar

java.lang.IllegalArgumentException: requirement failed: Ivy settings file /path/to/ivySettings.xml does not exist
	at scala.Predef$.require(Predef.scala:281)
	at org.apache.spark.deploy.SparkSubmitUtils$.loadIvySettings(SparkSubmit.scala:1331)
	at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:176)
	at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:156)
	at org.apache.spark.sql.internal.SessionResourceLoader.resolveJars(SessionState.scala:166)
	at org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:133)
	at org.apache.spark.sql.execution.command.AddJarCommand.run(resources.scala:40)
 

We should ship the ivySettings file to the driver so that addJar is able to find it.

Issue Links

• is related to: SPARK-34506 ADD JAR with ivy coordinates should be compatible with Hive transitive behavior
• is related to: SPARK-35072 spark.jars.ivysettings should support local:// and hdfs:// schemes
• relates to: SPARK-35073 SparkContext.addJar with an ivy path should not fail with a custom ivySettings file in non-YARN cluster modes
• relates to: SPARK-35074 spark.jars.xxx configs should be moved to config/package.scala
• links to: [Github] Pull Request #31591 (shardulm94)
• links to: [Github] Pull Request #31591 (shardulm94)