Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19707

Improve the invalid path check for sc.addJar

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.1.1, 2.2.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Currently in Spark there're two issues when we add jars with invalid path:

      • If the jar path is a empty string {--jar ",dummy.jar"}, then Spark will resolve it to the current directory path and add to classpath / file server, which is unwanted.
      • If the jar path is a invalid path (file doesn't exist), file server doesn't check this and will still added file server, the exception will be thrown until job is running. This local path could be checked immediately, no need to wait until task running. We have similar check in addFile, but lacks similar one in addJar.

        Attachments

          Activity

            People

            • Assignee:
              jerryshao Saisai Shao
              Reporter:
              jerryshao Saisai Shao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: