[SPARK-24195] sc.addFile for local:/ path is broken

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.3.1, 1.4.1, 1.5.2, 1.6.3, 2.0.2, 2.1.2, 2.2.1, 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Spark Core
• Labels: starter
• Assignee: Yuanjian Li
• Reporter: Felix Cheung
• Created: 07/May/18 05:12
• Updated: 20/Jul/18 03:27
• Resolved: 20/Jul/18 03:26

Description

In changing SPARK-6300
https://github.com/apache/spark/commit/00e730b94cba1202a73af1e2476ff5a44af4b6b2

essentially the change to
new File(path).getCanonicalFile.toURI.toString

breaks when path is local:, as java.io.File doesn't handle it.

eg.

new File("local:///home/user/demo/logger.config").getCanonicalFile.toURI.toString
res1: String = file:/user/anotheruser/local:/home/user/demo/logger.config

Issue Links

• links to: [Github] Pull Request #21533 (xuanyuanking)