[SPARK-18357] YARN --files/--archives broke

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.0, 2.2.0
• Component/s: Spark Core, YARN
• Target Version/s: 2.1.0
• Assignee: Kishor Patil
• Reporter: Thomas Graves
• Created: 08/Nov/16 15:43
• Updated: 17/May/20 18:13
• Resolved: 08/Nov/16 18:14

Description

SPARK-18099 broke --files and --archives options. The check should be ==null instead of !=:

if (localizedPath != null)

{ + throw new IllegalArgumentException(s"Attempt to add ($file) multiple times" + + " to the distributed cache.") + }

Issue Links

• is broken by: SPARK-18099 Spark distributed cache should throw exception if same file is specified to dropped in --files --archives
• links to: [Github] Pull Request #15810 (kishorvpatil)