Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22585

Url encoding of jar path expected?

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      I am calling

      sparkContext.addJar

      method with path to a local jar I want to add. Example:

      /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar

      . As a result I get an exception saying

      Failed to add /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark environment. Stacktrace:
      java.io.FileNotFoundException: Jar /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found
      

      Important part to notice here is that colon character is url encoded in path I want to use but exception is complaining about path in decoded form. This is caused by this line of code from implementation (see here):

      case null | "file" => addJarFile(new File(uri.getPath))
      

      It uses getPath method of java.net.URI which url decodes the path. I believe method getRawPath should be used here which keeps path string in original form.

      I tend to see this as a bug since I want to use my dependencies resolved from artifactory with port directly. Is there some specific reason for this or can we fix this?

      Thanks

        Attachments

          Activity

            People

            • Assignee:
              dubovsky Jakub Dubovsky
              Reporter:
              dubovsky Jakub Dubovsky
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: