[SPARK-1921] Allow duplicate jar files among the app jar and secondary jars in yarn-cluster mode

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 1.0.0
• Component/s: Deploy
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Xiangrui Meng
• Created: 25/May/14 06:41
• Updated: 21/May/19 05:36
• Resolved: 21/May/19 05:36

Description

In yarn-cluster mode, jars are uploaded to a staging folder on hdfs. If there are duplicates among the app jar and secondary jars, there will be overwrites that cause inconsistent timestamps. I saw the following message:

Application application_1400965808642_0021 failed 2 times due to AM Container for appattempt_1400965808642_0021_000002 exited with  exitCode: -1000 due to: Resource hdfs://localhost.localdomain:8020/user/cloudera/.sparkStaging/application_1400965808642_0021/app_2.10-0.1.jar changed on src filesystem (expected 1400998721965, was 1400998723123

Tested on a CDH-5 quickstart VM.