[SPARK-22585] Url encoding of jar path expected?

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core
• Assignee: Jakub Dubovsky
• Reporter: Jakub Dubovsky
• Created: 22/Nov/17 20:52
• Updated: 12/Dec/22 18:10
• Resolved: 30/Nov/17 01:25

Description

I am calling

sparkContext.addJar

method with path to a local jar I want to add. Example:

/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar

. As a result I get an exception saying

Failed to add /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark environment. Stacktrace:
java.io.FileNotFoundException: Jar /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found

Important part to notice here is that colon character is url encoded in path I want to use but exception is complaining about path in decoded form. This is caused by this line of code from implementation (see here):

case null | "file" => addJarFile(new File(uri.getPath))

It uses getPath method of java.net.URI which url decodes the path. I believe method getRawPath should be used here which keeps path string in original form.

I tend to see this as a bug since I want to use my dependencies resolved from artifactory with port directly. Is there some specific reason for this or can we fix this?

Thanks

Issue Links

• links to: [Github] Pull Request #19834 (james64)