[SPARK-18099] Spark distributed cache should throw exception if same file is specified to dropped in --files --archives

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.0.1
• Fix Version/s: 2.1.0, 2.2.0
• Component/s: Spark Core, YARN
• Assignee: Kishor Patil
• Reporter: Kishor Patil
• Created: 25/Oct/16 18:10
• Updated: 17/May/20 18:13
• Resolved: 03/Nov/16 21:13

Description

Recently, for the changes to SPARK-14423 Handle jar conflict issue when uploading to distributed cache
If by default yarn#client will upload all the --files and --archives in assembly to HDFS staging folder. It should throw if file appears in both --files and --archives exception to know whether uncompress or leave the file compressed.

Issue Links

• breaks: SPARK-18357 YARN --files/--archives broke
• links to: [Github] Pull Request #15627 (kishorvpatil)