[YARN-8562] Spark Application fails due to resource file timestamp difference

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.9.1, 2.8.4
• Component/s: yarn

Environment

Spark 2.3.1 (build with Hadoop 2.7)

Hadoop 2.8.4 & 2.9.1

• Assignee: Unassigned
• Reporter: Gao Pan
• Created: 22/Jul/18 04:00
• Updated: 22/Jul/18 04:00

Description

Application was submitted to YARN by spark-submit and accepted but alway failed with following exception. The expected timestamp alway comes with 000 as the right most digits and other digits always match.

2018-07-21 18:16:16 INFO Client:54 -
client token: N/A
diagnostics: Application application_1532213527145_0003 failed 2 times due to AM Container for appattempt_1532213527145_0003_000002 exited with exitCode: -1000
Failing this attempt.Diagnostics: [2018-07-21 18:16:15.668]Resource file:/Users/gpan/.sparkStaging/application_1532213527145_0003/com.amazonaws_aws-java-sdk-kms-1.11.76.jar changed on src filesystem (expected 1532222173000, was 1532222173543
java.io.IOException: Resource file:/Users/gpan/.sparkStaging/application_1532213527145_0003/com.amazonaws_aws-java-sdk-kms-1.11.76.jar changed on src filesystem (expected 1532222173000, was 1532222173543
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:255)