[SPARK-34225] Jars or file paths which contain spaces are generating FileNotFoundException exception

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.1.2, 3.2.0
• Component/s: Spark Core, Spark Submit, Windows
• Assignee: Kousuke Saruta
• Reporter: Lucian Timar
• Created: 25/Jan/21 13:26
• Updated: 14/Apr/21 08:40
• Resolved: 22/Mar/21 05:10

Description

 Whenever I try to use jars or files whose paths contain spaces FileNotFoundException exception is generated

just run spark-shell command having spaces in paths

See below

c:>spark-shell --files "c:\Program Files\...\myjar.jar"
or
c:>spark-shell --jars"c:\Program Files\...\myjar.jar"
or 
c:>spark-shell --conf spark.jars="c:\Program Files\...\myjar.jar"

any combination produce the same exception

 java.io.FileNotFoundException: Jar c:\Program%20Files\........ not found
        at org.apache.spark.SparkContext.addLocalJarFile$1(SparkContext.scala:1833)
        at org.apache.spark.SparkContext.addJar(SparkContext.scala:1887)
        at org.apache.spark.SparkContext.$anonfun$new$11(SparkContext.scala:490)
        at org.apache.spark.SparkContext.$anonfun$new$11$adapted(SparkContext.scala:490)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:490)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)
        at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)

I have noticed that the following function is causing the issue.

def addJar(path: String): Unit = {
...
} else {
  val uri = new Path(path).toUri
  

the path as the string is 

"file:///C:/Program%20Files/Nokia/....jar"

this call generates the following uri

"file:///C:/Program*%25*20Files/Nokia/....jar"

which results in an invalid file name.

Using 

val uri = Utils.resolveURI(path)

seems to resolve the issue.

 Until a fix is provided are there any workarounds to overcome this issue?

 

Issue Links

• links to: [Github] Pull Request #31718 (sarutak)
• links to: [Github] Pull Request #32164 (sarutak)