[SPARK-19707] Improve the invalid path check for sc.addJar

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: Spark Core
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 23/Feb/17 09:14
• Updated: 15/May/17 15:58
• Resolved: 24/Feb/17 17:29

Description

Currently in Spark there're two issues when we add jars with invalid path:

• If the jar path is a empty string {--jar ",dummy.jar"}, then Spark will resolve it to the current directory path and add to classpath / file server, which is unwanted.
• If the jar path is a invalid path (file doesn't exist), file server doesn't check this and will still added file server, the exception will be thrown until job is running. This local path could be checked immediately, no need to wait until task running. We have similar check in addFile, but lacks similar one in addJar.

Issue Links

• links to: [Github] Pull Request #17038 (jerryshao)
• links to: [Github] Pull Request #17987 (HyukjinKwon)