[SPARK-11595] "ADD JAR" doesn't work if the given path contains URL scheme like "file:/" and "hdfs:/"

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.2, 1.6.0
• Fix Version/s: 1.5.3, 1.6.0
• Component/s: SQL
• Target Version/s: 1.5.2, 1.6.0
• Assignee: Cheng Lian
• Reporter: Cheng Lian
• Created: 09/Nov/15 13:23
• Updated: 12/Nov/15 17:32
• Resolved: 09/Nov/15 22:33

Description

When handling ADD JAR, Spark constructs a java.io.File first using the input jar path, and then converts it into a URL (here). This works file for local file path without a URL scheme (e.g. /tmp/a.jar). However, java.io.File.toURI returns unexpected result when given a path containing a URL scheme (e.g. file:///tmp/a.jar or hdfs://host:9000/path/to/a.jar):

scala> new java.io.File("file:///tmp/file").toURI
res1: java.net.URI = file:/Users/lian/local/src/spark/workspace-a/file:/tmp/file

The consequence is that, although the ADD JAR command doesn't fail immediately, the jar is actually not added properly.

Issue Links

• relates to: SPARK-11191 [1.5] Can't create UDF's using hive thrift service
• links to: [Github] Pull Request #9569 (liancheng)
• links to: [Github] Pull Request #9570 (liancheng)